80-Muscle Simulated Human Walking: Relies on Exploration Noise During Training—Does It Fall When Turned Off?

A game animator shared on Reddit his experience training physically simulated NPC walking with reinforcement learning: based on the MyoSuite MyoLeg model (80 leg muscles) in MuJoCo/MuJoCo Warp, using the PPO algorithm (calling RSL-RL via mjlab) trained on a laptop GPU, with a reward function close to the work of Schumacher et al. 2023—walking at a target speed, adaptive effort cost, and "pain" penalties for joint limits and hard ground impacts. No motion capture demonstration data was used throughout.

His three attempts revealed a phenomenon worth attention in muscle-driven model training: the policy relies on its own exploration noise during training, and once the noise is turned off (using mean actions), it completely fails.

Three Attempts: From "Stomping" to "Stable Walking"

First attempt: After about 20 minutes of training, the policy learned to walk, but the gait was rough—peak foot forces were about 4.6 times body weight, with a cadence of about 230 steps per minute. More critically, when using mean actions (without noise), it fell every time.

Second attempt: Switched to sigmoid muscle commands and increased the impact cost. Foot forces dropped to about 2 times body weight, and the steps were smoother. But with noise turned off, the policy still fell within about 1.3 seconds.

Third attempt: While continuing training, slowly reduced the action noise standard deviation (eventually to 0.05). This time the policy could walk stably under mean actions: 32 tests with zero falls, cadence about 150 steps per minute, step length 0.44 meters, and foot forces about 1.5 times body weight. After that, it learned to follow speed and direction commands, start and stop (mostly), and drive a virtual human in UE5 in real time.

The current problem is gait asymmetry: stopping after a 90-degree left turn causes a fall, while turning right does not. The next round of training will add a mirror symmetry loss (Yu, Turk and Liu 2018).

MyoSuite and MyoLeg: MyoSuite is an open-source musculoskeletal model simulation environment from Meta (formerly Facebook), aimed at replacing traditional joint torque control with physiologically plausible muscle-driven models. MyoLeg is its lower-limb model, containing 80 muscles (corresponding to an 80-dimensional continuous action space), with each muscle controlled by an activation signal and dynamics solved by the MuJoCo physics engine. Unlike conventional joint-angle control, muscle-driven models have an extremely high-dimensional and highly redundant action space, and there is a nonlinear transmission from muscle activation to joint motion, so policy learning is significantly harder than ordinary humanoid robot control tasks.

PPO and RSL-RL: PPO (Proximal Policy Optimization) is a policy-gradient-based reinforcement learning algorithm that ensures training stability by limiting the magnitude of each policy update, and is widely used in continuous control tasks. RSL-RL is an open-source PPO implementation from ETH Zurich's Robotic Systems Lab, optimized for large-scale parallel simulation. mjlab is a training framework that connects MuJoCo (and its GPU-accelerated version MuJoCo Warp) with RSL-RL, enabling developers to run thousands of environment instances in parallel on a single GPU and thus complete training on laptop-level hardware.

Why Does the Policy Depend on Exploration Noise?

This is the core question the original post posed to those with experience in muscle model training: is it common for a policy to depend on its own exploration noise? Is the solution to reduce noise or use other methods (such as correlated noise Lattice)?

From the principles of reinforcement learning, exploration noise is supposed to help the policy explore the state space, not become part of the policy. When a policy becomes accustomed to noisy observations or action distributions during training, it may implicitly treat the noise as part of the input signal. This may be more prominent in high-dimensional continuous control tasks driven by muscles—with an 80-dimensional muscle action space plus complex dynamics, the policy easily finds "shortcuts" that depend on noise perturbations. By slowly annealing the noise standard deviation, the original author essentially lets the policy gradually adapt to a deterministic action distribution, ultimately remaining stable even under mean actions.

Action Noise and Mean Actions: In stochastic policy algorithms such as PPO, the policy network usually outputs the mean μ and standard deviation σ of a Gaussian distribution, and the actually executed action is sampled from N(μ, σ²). σ is the exploration noise, providing randomness during training to explore the state space; at inference time, the mean action is usually taken (i.e., σ=0) to obtain deterministic behavior. If the policy implicitly relies on the perturbation brought by σ during training—for example, treating noise as an extra "jitter" signal to maintain balance—then after turning off the noise, the behavior distribution shifts and the policy fails. Mitigation methods include: noise annealing (gradually reducing σ), using time-correlated noise (such as Lattice noise, making exploration smooth over time rather than independent and identically distributed), or fine-tuning with mean actions in the later stages of training.

Implications for Game NPC Animation

The animator's original intention was to let NPCs learn movement from physical simulation rather than relying on motion capture. Muscle-driven models can produce more natural physical interactions, but training stability remains a challenge. Three notable points:

• Reward design needs to balance speed and impact: The 4.6 times body weight impact force in the first attempt shows that pursuing speed alone leads to rough gaits, and adding impact cost significantly improves it.
• Noise scheduling is key to training stability: Slowly reducing the noise standard deviation lets the policy transition from random exploration to deterministic execution, avoiding the dilemma of "falling when noise is turned off."
• Symmetry issues require explicit constraints: The asymmetric behavior of stopping after left and right turns indicates that the policy may have learned a biased gait pattern, and mirror symmetry loss is a reasonable next step.

The author mentioned that he used Claude to help write a large amount of code and documented the entire process on a small YouTube channel. For developers attempting similar physics-driven character animation, this case provides a reproducible reference path: a complete workflow from high-dimensional muscle models and PPO training to noise annealing strategies.

Mirror Symmetry Loss: In character animation and motion control, human gait is naturally left-right symmetric. The mirror symmetry loss approach is: apply a left-right mirror transformation to the policy's state and action at a certain moment (swapping the joint and muscle dimensions corresponding to the left and right limbs, flipping lateral velocities, etc.), and require the mirrored action to be consistent with the action output by the policy for the mirrored state, thereby penalizing asymmetric behavior. Yu, Turk, and Liu systematically used such symmetry constraints in their 2018 work to improve motion quality. For an 80-dimensional muscle model, this loss requires correctly establishing the correspondence mapping between left and right muscles; otherwise, it will introduce incorrect constraints.

One-line Summary

A game animator trained 80-muscle NPC physical walking with PPO, using noise annealing to solve the problem of falling when noise is turned off.

Paragraph Summary

A game animator shared on Reddit the complete experience of training muscle-driven NPC walking with reinforcement learning. Based on the MyoSuite MyoLeg model (80 leg muscles), he trained with PPO (calling RSL-RL via mjlab) in MuJoCo/MuJoCo Warp on a laptop GPU, with reward design referencing the work of Schumacher et al. 2023, and without using motion capture data throughout. Three attempts showed: the first two policies fell after turning off exploration noise and switching to mean actions, while the third achieved zero falls in 32 tests under mean actions by slowly annealing the noise standard deviation (to 0.05), with a cadence of about 150 steps/min, step length 0.44 m, and foot impact force reduced from 4.6 times body weight to about 1.5 times body weight. There is still a left-right asymmetry problem of falling after stopping following a left turn; the author plans to introduce mirror symmetry loss, and has already been able to drive a UE5 virtual human in real time.