When Does Momentum Help in RLVR? Theory and Experiments Reveal Principles for Choosing the EMA Coefficient

Questioned Momentum: Empirical Contradictions in RLVR

Momentum is a classic tool for accelerating convex optimization and a standard component in modern large language model training. However, several recent empirical works have questioned the actual utility of first-order momentum in Reinforcement Learning with Verifiable Rewards (RLVR). This contradiction is not unfounded — the optimization landscape of RLVR differs fundamentally from supervised learning: reward signals are sparse and discrete, gradient estimates come from sampled responses, and variance is significantly higher than in traditional training scenarios.

So under what conditions can momentum truly accelerate RLVR? A new study provides a theoretical answer through a simplified bandit model and validates it on the Qwen series models.

RLVR (Reinforcement Learning with Verifiable Rewards) is a mainstream paradigm for post-training large language models in recent years. Its core characteristic is that reward signals can be programmatically verified, rather than relying on human annotation or reward model scoring. Typical tasks include mathematical reasoning (answers can be checked against a key), code generation (unit tests can be executed), and logic puzzles (results can be verified). Because rewards are discrete and sparse — the entire response sequence usually receives only a single 0/1 or graded signal at the end — policy gradient estimation must rely on multiple samples of the same prompt, resulting in variance far higher than gradients based on dense annotations in supervised learning. Additionally, algorithms commonly used in RLVR, such as GRPO (Group Relative Policy Optimization), construct relative advantages through intra-group response normalization, further altering the statistical structure of gradients. Understanding these characteristics is a prerequisite for understanding why momentum behaves differently in RLVR compared to traditional supervised training.

Theoretical Framework: Bandit Analysis of Bias-Corrected EMA Momentum

The researchers formalized RLVR as a bandit problem and adopted bias-corrected exponential moving average (EMA) momentum. The coefficient $\beta$ controls the decay rate of past gradient weights — the larger $\beta$ is, the more historical information is retained; when $\beta=0$, it degenerates to pure SGD.

The theoretical analysis reveals a key dividing line:

Exact Gradient Scenario: Increasing β Strictly Accelerates

When policy gradients can be computed exactly, the researchers prove that within the range avoiding overshoot, increasing $\beta$ strictly accelerates the optimization process. This aligns with intuition — under exact gradients, the accumulation of historical information helps smooth the optimization path and reduce variance.

Sampled Gradient Scenario: Noise Masks the Gain

In the real RLVR setting, gradients must be estimated from sampled responses. Here the situation becomes subtle: positive expected gains may be masked by sampling fluctuations. Theory shows that when the learning rate $\eta$ decreases, under a fixed $\beta$, the expected gain vanishes faster than the fluctuation itself — meaning that at small learning rates, the benefits of momentum are devoured by noise.

But the key insight is: if $\beta$ is simultaneously increased to keep $\eta/(1-\beta)$ within a safe range, then the non-vanishing gain will ultimately dominate the sampling noise. This yields an actionable principle: the learning rate and momentum coefficient must be adjusted in coordination, not chosen independently.

The bandit simplified model is widely adopted in theoretical analysis of reinforcement learning. Its approach is to reduce the sequential decision-making problem to single-step action selection, thereby stripping away the complexity of long-term credit assignment and focusing on the statistical properties of gradient estimation itself. In the bandit setting, policy gradient computation does not involve environment state transitions; each round only requires sampling from the action distribution and computing the gradient of the immediate reward, mathematically equivalent to a parameterized multi-armed bandit. The connection to RLVR lies in the fact that when each LLM response receives only a terminal reward and there are no intermediate rewards within the response, the noise structure of policy gradient estimation is indeed highly similar to a bandit. The limitations of this model are also apparent — it ignores the temporal correlations introduced by response length, token-level importance sampling corrections, and biases introduced by value function approximation. Therefore, generalizing theoretical conclusions to the full RLVR setting requires experimental validation as a safety net.

Experimental Validation: From Bandit to Qwen3/Qwen2.5

Bandit Simulation: Quantitative Match with Theoretical Predictions

The researchers conducted simulation experiments in the bandit setting, and the results quantitatively validated the above theoretical predictions. Configurations that kept $\eta/(1-\beta)$ constant indeed exhibited sustained advantages, while a fixed $\beta$ combined with a reduced $\eta$ showed vanishing gains.

Qwen3-4B-Base: SGD+EMA vs. AdamW Comparison

RLVR experiments on Qwen3-4B-Base further supported the theoretical principle. When using SGD+EMA and AdamW optimizers, configurations with larger momentum coefficients both showed advantages. This validates the applicability of the theoretical predictions in actual LLM post-training.

Qwen2.5-32B: A Simple β₁ Adjustment Brings Nearly 2 Percentage Points Improvement

The most compelling result came from the larger-scale Qwen2.5-32B model. Merely increasing AdamW's first-order momentum coefficient $\beta_1$ from its conventional value brought nearly 2 percentage points improvement in average validation accuracy. This improvement required no modification of any other hyperparameters, suggesting that the choice of momentum coefficient may have been long underestimated in RLVR.

AdamW's β₁ is the first-moment decay rate in the Adam family of optimizers, typically defaulting to 0.9, controlling the exponential moving average window of the first-order gradient. The larger β₁ is, the longer the historical gradient direction is retained and the smoother the optimization trajectory, but the slower the response to new gradient signals; the smaller β₁ is, the closer the optimizer behavior is to SGD with adaptive step sizes. The long-standing use of 0.9 in supervised learning is because training data is stationary and gradient signals are dense, requiring no excessive smoothing. But RLVR gradients are estimated from sampling, noisy, and non-stationary, so a larger β₁ can more effectively average out sampling fluctuations. Notably, there is a coupling between β₁ and the learning rate: AdamW's effective step size is approximately proportional to η/(1-β₁), so simply increasing β₁ is equivalent to implicitly amplifying the effective learning rate. This coupling relationship is the technical root of the original text's emphasis on "coordinated adjustment," and also explains why adjusting only β₁ can yield gains without modifying other hyperparameters.

Practical Implications: Re-examining Momentum Coefficient Selection

The core contribution of this work is providing theoretical guidance and experimental evidence for momentum selection in RLVR:

Learning rate and momentum must be designed in coordination. In traditional practice, $\beta$ is often treated as a fixed hyperparameter, but theory shows that the effective "momentum strength" is jointly determined by $\eta/(1-\beta)$. Increasing $\beta$ alone without adjusting $\eta$ may backfire.

Greater caution is needed in small learning rate scenarios. When policy gradient estimation noise is large, a reduced $\eta$ weakens the momentum gain. In this case, $\beta$ should be simultaneously increased to maintain effective momentum strength.

Substantial gains on large models. The nearly 2 percentage point improvement in validation accuracy on Qwen2.5-32B represents a significant improvement in RLVR post-training, and the implementation cost is extremely low — only one hyperparameter needs adjustment.

The researchers also note that the applicable boundaries of this principle still require validation across more tasks and model scales. The interaction between momentum and learning rate may differ across different RLVR algorithms (such as PPO, GRPO, etc.), warranting further exploration.

For engineers engaged in LLM post-training, the direct advice from this work is: when tuning RLVR, do not view $\beta$ in isolation. Treating $\eta/(1-\beta)$ as a whole for adjustment may be more effective than tuning them separately — at least in bandit theory and experiments on the Qwen series models, this principle has been validated.

One-line Summary

Theory proves that learning rate and momentum coefficient must be adjusted in coordination in RLVR; Qwen2.5-32B achieves nearly 2 percentage points improvement by adjusting only β₁.

Paragraph Summary

This study addresses the questioned utility of momentum in Reinforcement Learning with Verifiable Rewards (RLVR) by providing a theoretical explanation through a simplified bandit model, validated on the Qwen series models. Theoretical analysis shows that in the exact gradient scenario, increasing the bias-corrected EMA coefficient β strictly accelerates optimization; but in RLVR's real sampled gradient scenario, a fixed β combined with a reduced learning rate η causes expected gains to be masked by sampling noise. Only by simultaneously increasing β to keep η/(1-β) within a safe range can non-vanishing gains dominate the noise. Bandit simulations quantitatively validate this prediction, and experiments on Qwen3-4B-Base with SGD+EMA and AdamW both support this principle. On Qwen2.5-32B, merely increasing AdamW's β₁ brings nearly 2 percentage points improvement in validation accuracy. The practical recommendation is to adjust η/(1-β) as a whole rather than selecting β in isolation.