Cubic Doggo Upgrade: Training a Quadruped Robot with Horizontal Pushes

Reinforcement learning training for quadruped robots has made new progress. The Cubic Doggo project shared by the Reddit community showcases the latest training results: by introducing random forces in the horizontal direction, the robot's disturbance resistance has been significantly improved. The author not only released the training configuration and deployment code, but also summarized practical experience from simulation to real-world deployment.

Training Strategy Upgrade: From Curriculum Learning to Domain Randomization

The core of this update is the addition of horizontal force randomization during training—applying random forces of up to 5N to the 3kg robot to simulate lateral impacts in real environments. The red arrows in the video indicate the directions of these forces, and the trend in later training shows that the robot gradually learned to resist these disturbances.

The author adopted a curriculum learning schedule: initially only using a penalty term for joint position initialization, then gradually expanding the domain randomization range as training iterations progress, and tightening reward/penalty thresholds. This progressive strategy allows the robot to first master basic gait, then adapt to complex disturbances.

In addition, several new mechanisms were introduced in training: applying a penalty to torque distribution to encourage balanced force output across all four legs; penalizing foot slipping; applying low-pass filtering to actions; increasing the entropy coefficient after the curriculum schedule reaches its upper limit to enhance exploration. Also, the first few steps are skipped during training to avoid mesh interpenetration issues during simulation initialization.

Domain Randomization is a key technique for transferring reinforcement learning from simulation to reality. Its core idea is to randomize physical parameters in the simulation environment during training (such as mass, friction coefficients, motor delays, external disturbances, etc.), so that the policy learns to adapt to various conditions within the parameter distribution range, thereby narrowing the "Sim-to-Real" gap. Common randomization targets include ground friction, body mass, joint damping, control delay, and external forces. The randomization range needs to be carefully designed: too small a range leads to poor transfer, while too large a range makes training difficult to converge. It is usually combined with curriculum learning to gradually expand the range, consistent with the progressive strategy adopted in this article.

Curriculum Learning draws on the idea of gradual learning from easy to hard, allowing agents to master skills progressively. In robot reinforcement learning, curricula are typically reflected in reward thresholds, initial state distributions, disturbance intensity, or terrain complexity. For example, initially only penalizing joint position initialization deviations, then gradually introducing force disturbances and tightening penalty thresholds once the gait is stable. Curriculum schedule design requires monitoring training metrics (such as success rate, termination reasons) to avoid policy collapse caused by difficulty increasing too quickly. The author records curriculum schedule values and uses LLM to analyze trends, which is a practical engineering approach to ensuring smooth curriculum progression.

Entropy Coefficient is used in the reinforcement learning objective function to control the degree of policy exploration. Policy gradient methods (such as PPO) typically add an entropy regularization term to the loss, encouraging the policy to maintain randomness and avoid prematurely falling into local optima. Too large an entropy coefficient makes the policy too random and difficult to converge; too small leads to insufficient exploration and easy rigidity. Increasing the entropy coefficient after the curriculum learning reaches its upper limit is a "exploit first, explore later" strategy: low entropy in the early stage quickly masters basic gait, and high entropy in the later stage helps the policy find more robust solutions within the adapted disturbance range. This scheduling approach is relatively common in quadruped robot locomotion, especially when performance plateaus appear in later training stages—moderately increasing entropy can escape local optima.

Debugging Experience: Hidden Bugs and Logging System

The author shared several key lessons. Jitter during GUI training stems from parameter exploration and is normal; but zero-action tests must first be used to verify implementation correctness, especially when involving a large number of link indices. Even so, dangerous hidden bugs may still exist, such as accidentally added observation noise affecting control, which requires extensive logging to troubleshoot.

To this end, the project enhanced the logging system: recording termination reasons (showing that 20% of terminations are due to pitch angle exceeding 45 degrees, caused by left-right lateral forces), outputting curriculum schedule values, and saving data as .csv files that can be directly fed to LLM for trend analysis. This method of using LLM to assist in analyzing training curves provides a new approach for reinforcement learning debugging.

Deployment and Open Source: ONNX Export and Configuration Recording

After training is complete, the model is exported as an ONNX file for deployment. The author cleverly added a TRAIN_CONFIG field to the ONNX file, recording all training configurations for easy traceability. However, vector normalization parameters still need to be saved in a separate file, which is a minor regret.

The project is open-sourced on GitHub, including configuration files, training code, and deployment code. For developers who want to reproduce or improve upon it, these resources lower the barrier to entry.

ONNX (Open Neural Network Exchange) is an open model exchange format jointly launched by companies such as Microsoft and Facebook, designed to enable models trained in different deep learning frameworks (such as PyTorch, TensorFlow) to be deployed across platforms. ONNX serializes computational graphs into Protobuf format, supporting efficient execution on inference engines (such as ONNX Runtime, TensorRT). In robotics, ONNX is commonly used to export policy networks trained in simulation to onboard computing units (such as the Jetson series) for real-time inference. The metadata field of ONNX files allows developers to attach custom key-value pairs, and the author of this article uses this mechanism to embed TRAIN_CONFIG, achieving binding of training configuration with the model file for version traceability and reproducibility.

Vector normalization parameters are details easily overlooked in reinforcement learning deployment. During training, observation vectors are typically normalized using mean-variance normalization (such as VecNormalize) to stabilize network inputs. When exporting the model, if the normalization layer is not solidified into the computational graph, the inference side needs to separately save and load these statistics (mean, variance), otherwise observation distribution shift will cause policy failure. The article mentions that "vector normalization parameters still need to be saved in a separate file," which reflects this engineering pain point. The ideal approach is to export the normalization layer as part of the model, or serialize it together in the ONNX metadata.

Multi-source Perspectives: Community Feedback and Insights

Although the current main information comes from the original Reddit post, the comment section discussion (presumably) may involve the choice of domain randomization strength, design details of curriculum learning, etc. The author's randomization range for horizontal forces (5N for a 3kg robot) was carefully considered, providing a reference for other quadruped robot developers.

Overall, the iteration of Cubic Doggo demonstrates practical techniques of reinforcement learning in robot control: from curriculum learning to domain randomization, from hidden bug troubleshooting to LLM-assisted analysis, each step reflects the accumulation of engineering practice. With the code open-sourced, more developers can explore more robust controllers on this basis.