Bipedal Robot Sim-to-Real Transfer: A Full Account of the Challenges

A developer shared the latest progress on their bipedal walking robot with the community. Behind this seemingly straightforward update lies one of the most central and difficult problems in reinforcement learning robotics — the Sim-to-Real transfer gap. Their documentation not only showcases incremental results, but also openly discusses the engineering struggles that are hard to articulate from the front lines of development.

Progress After Upgrading to Stronger Motors

The policy in this update was trained in simulation using a more powerful motor model. The developer admits that at first glance, the movements don't look noticeably smoother — but compared to the previous version, even just the "standing" behavior is visibly cleaner and more stable.

What truly satisfied him was the walking performance. When commanding the robot to slowly move forward and backward, there was almost no extraneous motion or jerking. For a bipedal system built from scratch, achieving low-jitter slow-speed gait is a milestone worth documenting. The improved motor performance played a key role here — greater actuator torque gave the policy more control headroom, reducing the compensatory jerking caused by insufficient power.

The Mystery of Smooth Simulation, Degraded Reality

The most thought-provoking aspect of the update is a question the developer raised that remains unanswered: Why does forward walking on the real robot perform noticeably worse than in simulation?

This is a classic symptom of Sim-to-Real transfer failure. To reproduce the problem, he conducted extensive troubleshooting: manually adding joint angle offsets to the trained policy, reducing actuator torque, and introducing various mismatches between simulation and reality. Yet none of these perturbations could reproduce the "bad" behavior seen on the real machine within simulation.

This suggests the root cause may not lie in the common error sources he had already considered. The anomalous behavior on the real robot could stem from more subtle factors — sensor latency, communication timing, unmodeled mechanical compliance, or nonlinear motor responses. Without being able to reproduce it in simulation, he has no way to specifically target it with domain randomization during training, leaving him temporarily without "a good explanation" or any clear idea of what to change in the training setup.

Domain Randomization is currently one of the most mainstream techniques for bridging the Sim-to-Real gap. The core idea is to intentionally randomize various physical parameters during training — such as mass, friction coefficients, motor delays, and sensor noise — forcing the policy to learn stable behavior across a broad distribution of parameters. When deployed on the real robot, even if the actual parameters deviate from the training mean, the policy has sufficient robustness to cope. However, domain randomization has a fundamental limitation: it can only cover error sources that the developer has already identified and explicitly modeled. Once the real robot exhibits behavior stemming from factors the developer never thought to randomize — such as nonlinear vibrations from a specific mechanical structure or the unique dead-zone characteristics of a particular motor brand — the policy will expose its training blind spots. This is precisely why "unable to reproduce in simulation" puts developers in such a difficult position: without reproduction, there's no targeted training signal, and domain randomization has nothing to work with.

Comparing Three Simulators: Genesis, MuJoCo, and Isaac Lab

The developer's toolchain choices are worth referencing for similar projects. His policy was trained in Genesis and tested via sim-to-sim in MuJoCo, with good results. Sim-to-sim validation is a common practice — using a different simulator from the training environment to check policy robustness, which can predict real-world generalization to some degree.

The problem emerged with Isaac Lab. He encountered a stubborn contact/friction issue: no matter how he tuned it, he couldn't find realistic foot-ground friction. With low values, the robot slid around as if on ice; with high values, it was as though the feet were glued to the ground. He was never able to find a natural middle ground in Isaac Lab that resembled the behavior in Genesis, MuJoCo, or the real robot.

This detail reveals an often-overlooked reality: different physics engines implement contact models very differently. Friction and contact solving are the hardest parts to tune in rigid-body simulation, and the same parameter set often cannot be directly transferred between engines. The developer also posted publicly in the community asking if anyone had encountered similar issues in Isaac Lab — this kind of cross-engine experience sharing is exactly where the value of the open-source robotics community lies.

Contact and friction solving in rigid-body simulation is fundamentally a numerical optimization problem, and different physics engines employ significantly different underlying algorithms. MuJoCo uses a soft-constraint contact model that approximates penetration through penalty terms, making parameter tuning relatively intuitive. Isaac Lab is based on NVIDIA's GPU-parallel PhysX engine, whose contact solver is specially optimized for large-scale parallel simulation and behaves systematically differently from solvers that prioritize single-instance accuracy. Genesis, as a newer simulation framework, also has its own specific contact handling implementation. This means that "friction coefficient = 0.8" can correspond to completely different actual contact behavior across the three engines, and parameters cannot be directly reused across them. Sim-to-sim testing (cross-validating across multiple simulators) leverages exactly this difference to evaluate policy robustness — if a policy performs stably across multiple simulators with very different contact models, the success rate of transferring to a real robot is generally also higher.

Why the Next Update Won't Come Quickly

The developer is measured about his iteration pace. He explicitly stated he doesn't want to churn out a stream of small, incremental updates — and the robot is still far from performing anything resembling a "trick."

The more practical constraint is the slow pace of iteration itself. The full cycle for teaching the robot a new behavior is: design a reward function in simulation, train the policy, then test on the real robot. And real-robot testing carries inherent risk — as seen in the video from this update, a single test run has "a significant probability of ending with some limb or joint getting damaged."

Hardware wear and tear is an unavoidable cost in both amateur and professional robotics development. Each failed real-world experiment consumes not only time, but also potentially means replacing parts and reassembling the robot. This indirectly explains why reinforcement learning robotics research relies so heavily on simulation — the cost of trial and error in the real world is simply too high.

Reward function design (Reward Shaping) is one of the most challenging and experience-dependent aspects of reinforcement learning robotics development. For legged locomotion control, the reward function typically needs to simultaneously encode multiple competing objectives: velocity tracking, energy efficiency, postural stability, contact force constraints, and more. The weighting of each reward term directly determines the emergent gait style — a slight imbalance in weights can lead the policy to learn "strange" behaviors that accomplish the task in simulation but are completely infeasible on the real robot, such as high-frequency oscillation to avoid a particular penalty term. From simulation training to real-robot validation to identifying problems and tracing back to revise the reward function, each complete iteration cycle can take days or even weeks. And every failed real-robot test can simultaneously cost debugging time and hardware parts, making the engineering pace of legged robot reinforcement learning far heavier than pure software development.

What This Case Study Tells Us

This unassuming progress post distills several genuine pain points in current embodied AI development: there will always be a gap between simulation and reality that cannot be fully eliminated; physics engine contact modeling remains a fragile link; and hardware iteration is constrained by cost and wear. For any developer looking to get started with legged robot reinforcement learning, these hard-won lessons from the front lines are far more valuable than polished success videos.