Training a Hexapod Robot with Pure PPO: No Keyframes, Real-Time Browser Interaction

Developer trains a hexapod robot with PPO in MuJoCo and brings physics simulation and policy inference into the browser for real-time interaction.
This article introduces a hexapod robot project based entirely on PPO reinforcement learning: all 24 joints are driven by a policy network, with no keyframes or motion capture, and the robot can walk on randomized terrain and autonomously recover balance after impacts. Physics simulation runs at 480 Hz, the policy outputs actions at 30 Hz, and observations include body state, joint state, commanded velocity, and foot height maps. The developer summarizes three key lessons: physics setup affects gait as much as the reward function; reward curves can be misleading and require video validation; sim-to-browser transfer requires consistency between training, validation, and browser simulations. The browser side uses MuJoCo WASM, ONNX Runtime Web, and three.js, and users can throw a 171 kg steel ball to test robustness. The article concludes with an open question about how to handle data when physics explosions cause episode termination.
In reinforcement learning, using keyframes and motion capture to drive virtual characters is common practice, but a developer on Reddit took a different approach—relying entirely on the PPO algorithm to train a hexapod robot in the MuJoCo physics engine to learn to walk, and even autonomously recover balance when struck. More excitingly, the entire physics simulation and policy inference run in the browser, allowing you to interact with the robot in real time and even throw a 171 kg steel ball at it.

No Keyframes, No Motion Capture: Pure Reinforcement Learning Drives 24 Joints
This hexapod robot has 24 joint motors, and all movements come from a policy network trained with PPO, without any manually designed animations or reference motions. The developer emphasizes that the staggering recovery posture after being hit also emerged naturally during training. This means the virtual robot's behavior is entirely shaped by physical interactions and reward functions, rather than being scripted.
In terms of technical implementation, MuJoCo runs the physics simulation at 480 Hz, while the policy network outputs actions at 30 Hz. The policy network's observations include: body state, joint state, commanded velocity and orientation, and a height map of the ground around the feet. The action space is the target angles for the 24 joints; the motors track these angles, and the physics engine handles the subsequent motion. The reward function is designed simply: tracking commanded velocity, staying upright, maintaining correct orientation, saving energy, and ensuring only the feet contact the ground. Training is performed on randomized block terrain, with random pushes and crates as obstacles added.
PPO (Proximal Policy Optimization) is a reinforcement learning algorithm based on policy gradients. Its core idea is to limit the difference between the new and old policies at each update by clipping the importance sampling ratio to avoid excessively large policy updates, thereby ensuring training stability. PPO belongs to the actor-critic architecture, where the actor network outputs the action probability distribution and the critic network estimates the state value, and the two are optimized jointly. Compared to earlier algorithms like TRPO, PPO is simple to implement, computationally efficient, and performs excellently in continuous control tasks, having become one of the default choices for training robot policies in physics simulation environments such as MuJoCo. In this article, the entire joint control policy of the hexapod robot was trained from scratch using PPO.
MuJoCo (Multi-Joint dynamics with Contact) is a high-performance physics engine focused on contact dynamics, widely used in robotics, biomechanics, and reinforcement learning research. It is known for its computational efficiency and accurate contact models, capable of rapidly simulating complex interactions of multi-joint systems. In reinforcement learning training, MuJoCo is often used as the underlying engine for standard simulation environments (such as OpenAI Gym's MuJoCo task suite). In this article, MuJoCo runs the physics simulation at 480 Hz, much higher than the policy network's 30 Hz action output frequency. This design allows the physics engine to perform multiple fine integrations between policy decisions, thereby ensuring the stability of contact and collision calculations.
Three Key Lessons: Physics Setup, Reward Curves, and Sim-to-Browser
The developer shared three valuable lessons that are highly relevant for those training in fast simulators.
First, the physics setup affects gait as much as the reward function. Joint stiffness, damping, contact parameters, and time step can cause the same network to exhibit vastly different robot behaviors. The developer encountered an issue where feet got stuck on block edges; improving collision shapes was ineffective, and the problem was ultimately solved by reducing the time step.
Second, reward curves can "lie." Some policies drag a foot or leave a leg idle yet still achieve decent reward scores. Therefore, the developer recorded videos of each candidate policy at real speed and judged quality visually—this proved more reliable than just looking at rewards.
Third, sim-to-browser is a real transfer problem. The fast training simulator, the more accurate simulator used for validation, and the WebAssembly build in the browser must remain consistent; otherwise, a policy that walks normally during training will fall over on screen.
On the browser side, the developer used the official MuJoCo WebAssembly build, ONNX Runtime Web, and three.js. Both physics and policy run locally, with no server required.
Sim-to-browser transfer refers to the process of deploying a trained policy from the training environment to run in the browser, and is a variant of the broader sim-to-real transfer problem. In traditional sim-to-real, the discrepancy mainly comes from modeling errors between simulation and real physics; in the sim-to-browser scenario, the discrepancy arises from numerical precision, time step, and contact solver differences between different simulator implementations (fast training simulator, accurate validation simulator, WebAssembly build). WebAssembly (WASM) is a binary instruction format that can run efficiently in browsers; MuJoCo officially provides a WASM build, enabling physics simulation to execute locally on the client side. ONNX Runtime Web is responsible for loading and inferring the ONNX-format policy network in the browser. All three must remain consistent; otherwise, the policy will behave very differently on screen compared to training.
Interactive Demo: Throw a Steel Ball at the Robot
The most appealing aspect of this project is real-time interaction. You can open the demo in your browser via the link and throw a 171 kg steel ball or a crate at the robot to observe how it handles the impact and tries to regain balance. This interactivity is not only fun but also intuitively demonstrates the robustness of the policy.
The developer admits that the robot still gets stuck in some terrain depressions, and the gait is not yet fully natural. But these issues are precisely the directions for future improvement.
Open Question: How to Handle Physics Explosions
At the end of the post, the developer raised an open question: in a fast simulator, when the physics simulation "explodes" (numerical instability) causing an episode to end, how should that data be handled? Should it be treated as a terminal state, truncated with bootstrapping, or simply discarded from the batch? The developer mentioned that one of these choices once caused a previously improving run to collapse. This question sparked community discussion and highlights the importance of handling abnormal terminations in reinforcement learning training.
This project is built on MuJoCo's WASM build and ONNX Runtime Web, bringing physics simulation and policy inference directly to the browser and providing an intuitive demonstration platform for robot learning. Its open-source spirit and experience sharing are of positive significance to the reinforcement learning community.
(Note: The technical implementation and experiences mentioned in this article are all from the original Reddit post and the developer's blog, with no additional data added.)
In reinforcement learning training, episode termination is typically divided into two categories: termination and truncation. Termination refers to the environment reaching a true terminal state (e.g., the robot falls), in which case the value function target should be zero; truncation refers to artificially ending an episode due to time limits or numerical anomalies, in which case the target should be corrected using the current value estimate via bootstrapping, rather than being set to zero. Physics explosions are abnormal truncations caused by numerical instability; if mistakenly treated as terminal states, the value network will learn incorrect low-value estimates, potentially causing training collapse. Common handling methods include: discarding the episode data, truncating and bootstrapping, or applying a penalty to the explosion state and marking it as terminal. The developer's mention that "one choice caused the run to collapse" is likely related to improper handling of value targets.
One-line Summary
Developer trains a hexapod robot with PPO in MuJoCo and brings physics simulation and policy inference into the browser for real-time interaction.
Paragraph Summary
This article introduces a hexapod robot project based entirely on PPO reinforcement learning: all 24 joints are driven by a policy network, with no keyframes or motion capture, and the robot can walk on randomized terrain and autonomously recover balance after impacts. Physics simulation runs at 480 Hz, the policy outputs actions at 30 Hz, and observations include body state, joint state, commanded velocity, and foot height maps. The developer summarizes three key lessons: physics setup affects gait as much as the reward function; reward curves can be misleading and require video validation; sim-to-browser transfer requires consistency between training, validation, and browser simulations. The browser side uses MuJoCo WASM, ONNX Runtime Web, and three.js, and users can throw a 171 kg steel ball to test robustness. The article concludes with an open question about how to handle data when physics explosions cause episode termination.
Related articles

Gemini 4 Hydrogen Hands-On: Google's Fastest and Cheapest AI Model
Hands-on with Gemini 4 Hydrogen: the fastest, cheapest model in the Gemini 4 family. We break down the Carbon/Argon/Hydrogen split, API pricing traps, free-tier limits, and the cost trade-offs of four thinking levels—with a real bug-debugging demo.

Gemini 4 Hydrogen vs Sonnet Comparison Test: Uses Only 1% of Weekly Quota, Performance on Par
Hands-on test: Gemini 4 Hydrogen vs Sonnet on the same CRUD task—Hydrogen used only 1% weekly quota with comparable performance and better UI details, while Anthropic's model used 7%. Developer says daily tasks don't require expensive models.

Gemini 4 Neon Released: Beats Opus 5.5 on Multiple Benchmarks, Google Returns to the Top Tier
Google DeepMind releases Gemini 4 Neon: 61 on the Artificial Analysis intelligence index (3rd) and 1st on the coding index, beating Opus 5.5 on multiple benchmarks. $3/$15 per million tokens; available in the Gemini API and AI Studio.