Sim2Real in Practice: A Complete Guide to Reinforcement Learning Training and Deployment for a Two-Wheeled Balancing Robot

From Simulation to Reality: A Technical Breakthrough in a DIY Robot Project

In recent years, Sim2Real (Simulation-to-Reality transfer) technology has been gaining significant traction in the robotics field. Sim2Real is one of the core research directions in robotics and reinforcement learning. The fundamental idea is to train an agent's control policy in a virtual simulation environment, then deploy the trained policy directly onto a real physical robot. The key advantage of this approach is that it avoids the high cost and safety risks of real-world data collection — in simulation, a robot can fall over millions of times without damaging any hardware. However, the biggest challenge facing Sim2Real is the "Reality Gap": simulators cannot perfectly replicate the physical properties of the real world, including friction, contact dynamics, sensor noise, and more. To narrow this gap, researchers typically employ techniques such as Domain Randomization and System Identification, deliberately introducing parameter perturbations during training to force the policy to learn robust responses to uncertainty.

A developer from the Reddit community recently shared an impressive project — a height-adjustable two-wheeled balancing robot trained entirely on 100% synthetic data, with successful policy transfer from simulation to the real physical world.

This project offers substantial reference value in both technical depth and engineering implementation, particularly for researchers and engineers focused on embodied intelligence and real-world reinforcement learning deployment. The details are well worth a deep dive.

Hardware Architecture: Compact Without Compromise

Joint Design and Actuator Selection

Unlike traditional two-wheeled balancing robots that rely solely on two drive wheels, this robot features 3 joints per leg, forming a complete hip-knee-wheel structure. This design grants the robot three key capabilities: maintaining balance at different heights, independent tilting of left and right legs, and keeping the chassis level during acceleration.

For actuators, the project uses six Xiaomi Cybergear QDD (Quasi-Direct Drive) motors. QDD motors are a class of key actuator solutions that have emerged in legged robotics in recent years. Unlike traditional high-reduction-ratio servo motors, QDD motors typically use a relatively low gear ratio (generally between 6:1 and 9:1), giving them excellent backdrivability — meaning external forces can push the motor shaft with relative ease. This property is crucial for legged robots: when the robot's foot collides with the ground, the low-impedance joint can absorb the impact like a spring, rather than experiencing a rigid collision. The Xiaomi Cybergear series integrates a brushless DC motor, planetary gearbox, encoder, and FOC driver into a single unit, supporting position, velocity, and torque control modes. Due to their high cost-performance ratio, they have been widely adopted in the open-source robotics community. A 6s LiPo battery provides power, with the overall power density and weight ratio kept at reasonable levels.

Communication Architecture and Sensing Approach

The controller is an ESP32, a low-cost embedded platform popular in the DIY robotics community. Notably, the project uses two CAN buses simultaneously: one implemented through the ESP32's native TWAI interface, and another via SPI expansion.

CAN (Controller Area Network) bus was originally developed by Bosch for automotive electronic systems. Thanks to its high reliability, multi-node support, and differential signal noise immunity, it has become one of the mainstream solutions for multi-joint robot communication. On a CAN bus, all nodes share the same bus and resolve conflicts through a priority-based arbitration mechanism. However, a single CAN bus has limited bandwidth (1 Mbps for classic CAN), and when multiple high-frequency communication nodes are connected, excessive bus load can lead to message delays or losses. In this project, all six motors need to complete command transmission (position/torque) and state feedback reading within each control cycle (typically 1–5 ms), making the communication pressure on a single bus obvious. The dual CAN bus approach, with each bus handling three motors, effectively alleviates the bandwidth bottleneck during concurrent communication of six motors, ensuring real-time performance of the control loop. The ESP32's TWAI interface is its built-in CAN controller, while a second CAN bus is achieved by connecting an external CAN controller chip (such as the MCP2515) via SPI.

The IMU currently uses an MPU6050, with plans to upgrade to a BNO086. An IMU (Inertial Measurement Unit) is the core sensor of a balancing robot, responsible for real-time measurement of angular velocity and acceleration to infer attitude angles. The MPU6050 is a classic 6-axis MEMS IMU chip integrating a 3-axis accelerometer and 3-axis gyroscope, widely used in the maker community due to its extremely low price. However, the MPU6050's raw data requires the main controller to perform attitude fusion calculations (such as Kalman filtering or complementary filtering), and its noise characteristics and zero-bias drift perform poorly in highly dynamic scenarios. In contrast, the BNO086 features Bosch's built-in intelligent sensor fusion algorithm (running on an on-chip coprocessor), which can directly output fused quaternion attitude data, along with advanced features like motion classification and vibration compensation. For dynamic balance control that depends on millisecond-accurate attitude feedback, upgrading the IMU can often deliver a qualitative leap in performance.

Reinforcement Learning Training Strategy: Sim2Real Transfer Driven by Pure Synthetic Data

Hybrid Joint and Task Space Training Method

The most core technical highlight of this project lies in its training methodology: 100% synthetic data, zero real-world samples. The policy is trained from scratch using mjlab (a MuJoCo wrapper tool), achieving convergence in approximately 4 hours on a single RTX 3080.

MuJoCo (Multi-Joint dynamics with Contact) is a physics simulation engine developed by Emo Todorov, renowned in the robotics and reinforcement learning communities for its efficient and accurate contact dynamics simulation capabilities. After DeepMind open-sourced MuJoCo in 2022, its ecosystem expanded rapidly, making it one of the mainstream robot reinforcement learning simulation platforms alongside Isaac Gym. MuJoCo uses Generalized Coordinates representation and a semi-implicit Euler integration scheme, achieving extremely fast single-step simulation speeds while maintaining numerical stability. As a MuJoCo wrapper, mjlab provides a more user-friendly Python API, environment management, and training pipeline integration, lowering the engineering barrier to building training workflows from scratch. The ability to complete policy training in 4 hours on a single RTX 3080 is largely attributable to MuJoCo's computational efficiency and engineering optimizations in parallel environment sampling.

The training employs a Hybrid Joint and Task Space Training approach. In robot control theory, Joint Space and Task Space (also called Cartesian Space or Operational Space) are two fundamental control frameworks. Joint space control directly manipulates each joint's angle, angular velocity, or torque, with the advantage of naturally respecting physical constraints such as joint limits and singularities, but it is not intuitive when describing target behaviors of end effectors (such as toe tips or wheels). Task space control formulates control objectives directly at the level of the end effector's position, velocity, or force, aligning better with high-level task semantics (e.g., "maintain constant wheel ground contact force"), but requires inverse kinematics/inverse dynamics mapping back to the joint level, which can be problematic near joint limits. Hybrid approaches typically apply different control space strategies to different degrees of freedom or subtasks: for example, leg swing trajectories are planned in task space, while joint torque limits are enforced in joint space. This layered hybrid strategy is particularly effective in complex legged systems, preserving the advantages of each approach while helping the policy generalize better to the dynamics differences of real hardware.

Model Compression: 99.87% Size Reduction for Embedded Deployment

Targeting resource-constrained embedded platforms like the ESP32 (240 MHz clock, approximately 520 KB RAM), the author achieved an extreme compression of the collision model through Primitives Optimization, resulting in a 99.87% reduction in model size.

In physics simulation, collision detection is one of the most computationally expensive components. While precise 3D mesh collision models can accurately represent a robot's geometry, their collision detection algorithms (such as GJK, SAT, etc.) have computational complexity that increases dramatically with the number of vertices and faces. Geometric Primitives — spheres, capsules, cylinders, boxes, etc. — have analytical collision detection equations, with computational efficiency several orders of magnitude higher than mesh-based collision. For example, collision detection between two spheres requires only computing a single distance and comparing it to the sum of their radii. A 99.87% size reduction means the original model may have contained tens of thousands of triangular faces, while the compressed version requires only a few dozen primitive parameters for approximate representation, delivering fundamental improvements to both simulation speed and policy inference speed. In embedded deployment scenarios, replacing meshes with primitives is not merely a performance optimization but a prerequisite for deployment feasibility — while drastically reducing computational overhead, the loss in physical simulation accuracy is kept within acceptable bounds, which is also a critical factor enabling successful policy transfer to the real robot.

Current Progress and Technical Challenges

Initial Sim2Real Test Results

In the tests presented, the hip and knee joints were locked at startup (joint angles can be varied, but no active control was applied during operation), and the robot relied solely on wheel motors to maintain balance. The author candidly noted that the balancing performance "is starting to work, but isn't perfect yet" — an honest and expected assessment. Sim2Real transfer inherently faces dynamics deviations caused by simulator parameter errors (mass, inertia, friction coefficients, etc.), so getting the initial tests to run at all is already a positive signal.

It's worth noting that Domain Randomization plays a key role in narrowing the reality gap. This technique applies random perturbations to the simulation environment's physical parameters (such as mass, friction coefficients, motor delays, etc.) during training, enabling the policy to maintain a degree of robustness when facing the inevitable parameter deviations of the real world. The imperfect balance observed in the initial tests is likely related to insufficient coverage of the parameter space in simulation, and further improvements can be expected through more refined domain randomization tuning.

Next Steps: Full Joint Dynamic Control

The next phase of the project aims to unlock runtime control of the hip and knee joints, achieving true dynamic height adjustment and body tilting. This will significantly increase the dimensionality of the control problem — from the current 2 degrees of freedom (two wheels) to 6 degrees of freedom (hip, knee, and wheel, each times two), with both the state space and action space of the policy expanding dramatically. The increased dimensionality not only places higher demands on policy network capacity and training sample efficiency, but also means the simulator needs to more accurately model complex physical phenomena such as coupled inter-joint dynamics and contact state transitions. The IMU upgrade to the BNO086 is also critical preparation for this phase, as more accurate attitude feedback will provide a more reliable state estimation foundation for full-joint control.

Technical Significance and Developer Takeaways

The value of this project goes beyond being "just another balancing robot" — it demonstrates a reproducible, low-cost Sim2Real pathway: a consumer-grade GPU, open-source simulation framework, commercially available QDD motors, plus a DIY controller, are all that's needed to build a balancing robot platform with research-grade capabilities. The emergence of this pathway marks Sim2Real technology's transition from an exclusive tool of top-tier laboratories to the broader developer community — research that previously required equipment costing hundreds of thousands of dollars and professional teams can now be replicated by a single experienced individual developer.

For developers exploring the deployment of embodied intelligence, this case offers several practical takeaways:

• Model compression is a must-solve problem for embedded deployment: The 99.87% compression rate demonstrates the viability of geometric primitive approximation in motion control scenarios, and this method can be extended to real-time collision detection and physics reasoning tasks on other resource-constrained platforms
• Hybrid space training deserves close attention: In multi-joint legged systems, hybrid joint and task space training strategies effectively balance control precision and generalization capability, avoiding the inherent limitations of single-space frameworks
• Redundant dual CAN bus design should not be overlooked: This is a critical engineering detail for ensuring real-time multi-motor control, and any robot system with more than four CAN nodes should carefully consider bus topology planning

The author has opened an online policy demo (vertex.bot/robot), where interested readers can directly test the policy's actual performance, which also includes a policy for another robot called the Pollens MicroDuck.