Reward Budgeting Mechanism Solves Premature Convergence in LLM Reasoning RL

Reinforcement learning (RL) has shown great potential in improving the reasoning capabilities of large language models (LLMs), but a stubborn problem persists: the policy distribution becomes increasingly sharp as training progresses, causing the model to converge prematurely to suboptimal solutions and performance gains to stall. A recent study re-examines this phenomenon from the perspective of update dynamics and proposes a simple plug-in method called "reward budgeting" that significantly improves multiple RL baselines without explicitly maintaining entropy.

The Limitations of Entropy as a Control Signal

Previous research attempted to use policy entropy both as a diagnostic tool and as a control signal, artificially increasing entropy to delay convergence. However, the authors of this paper found in their experiments that merely raising entropy cannot reliably change the final plateau. This suggests that entropy may be a symptom rather than the cause—the real issue lies in how reward or advantage gaps are mapped to probability ratio gaps.

This observation prompted the researchers to adopt a more fundamental perspective: analyzing the implicit target policy of RL. They derived a "sharpness factor" κ (kappa) and its effective version κ_eff to describe the strength with which reward gaps or advantage gaps are converted into probability ratio gaps. The larger the κ value, the more easily the policy becomes sharp, prematurely locking into certain behavioral patterns.

Policy entropy in reinforcement learning measures the uncertainty or randomness of policy outputs. For LLMs, it roughly corresponds to the dispersion of different responses the model generates given a prompt: high entropy means the model tends to explore multiple reasoning paths, while low entropy means it is highly concentrated on a few or even a single path. During RL training, the policy typically undergoes entropy reduction as rewards are optimized, eventually "collapsing" into a certain fixed behavior—this is the typical manifestation of premature convergence. Therefore, many works treat entropy as a diagnostic indicator—entropy decline signals insufficient exploration; and also as a control signal—artificially maintaining randomness through entropy regularization terms or entropy bonuses. However, entropy is an aggregate statistic of the policy distribution and cannot directly reflect how reward gaps are translated into parameter update magnitudes, which explains why tuning entropy alone sometimes cannot change the plateau.

Reward Budgeting: Advantage Construction with a Fixed Total

Based on the above analysis, the researchers proposed the reward budgeting mechanism. Its core idea is very simple: assign a fixed total reward budget to each prompt, then construct the advantage function under this budget constraint. For binary rewards and independent on-policy rollouts, this unclipped score function estimator follows the gradient of an exact finite-G concave objective function with diminishing returns characteristics; meanwhile, as more rollouts become correct, the advantage gap decreases accordingly.

For general reward forms, under given regularity conditions, the method can be asymptotically interpreted as a shifted log-mean form. The researchers carefully note that these results are intended to motivate the clipped implementation rather than establish exact objective equivalence. This theoretical restraint actually enhances the credibility of the method.

On-policy rollout refers to sampling trajectories using the policy currently being trained, then using these trajectories to compute advantages and update the same policy. In contrast, off-policy methods use data generated by old or behavior policies and require importance sampling correction. On-policy guarantees unbiased gradient estimates, but at the cost of low data utilization and the need to resample after each update. The theoretical derivation in this paper is explicitly built on on-policy, independently sampled rollouts, which gives the conclusion that "under a fixed total reward budget, the advantage gap decreases as correct samples increase" a clear probabilistic meaning; once off-policy data or cross-iteration reuse is introduced, the statistical properties of advantages change, and the effectiveness of the budget mechanism needs to be re-examined.

Experimental Results: Consistent Improvements Across Models and Benchmarks

On two models, Qwen3-8B and Llama3.2-3B-Instruct, the reward budgeting mechanism consistently improved multiple RL baselines. Specifically, it achieved up to +12.50 improvement on the AMC23 benchmark, and +10.00 improvements on AIME24 and AIME25 respectively. These gains were obtained without explicitly preserving entropy, directly challenging the common assumption that "entropy must be maintained to avoid premature convergence."

Insights from the Update Dynamics Perspective

The value of this work lies not only in a plug-and-play technique but also in providing a new framework for understanding premature convergence. Rather than treating entropy as a knob that needs manual adjustment, it is better to focus on how reward signals shape update dynamics. Reward budgeting naturally produces diminishing returns by limiting the total reward per prompt, preventing the policy from becoming excessively sharp due to a few high-reward samples.

For practitioners, this means that when training LLM reasoning capabilities with RL, one can try adding this budget constraint at the advantage computation stage without modifying existing entropy regularization or clipping mechanisms. The code-level changes are minimal, but the potential benefits are significant.

Advantage function measures how much better a certain action (in LLMs, a token or an entire response) performs relative to the baseline policy's average, typically written as A(s,a)=Q(s,a)−V(s), where Q is the action value and V is the state value. Policy gradient methods update parameters by multiplying the advantage with the gradient of the log probability: a positive advantage increases the probability of that action, while a negative one decreases it. The absolute scale of the advantage directly determines the step size of each update, so when a few high-reward samples produce extremely large advantages, the policy is forcefully pushed toward these samples, and the distribution rapidly sharpens. Reward budgeting intervenes at exactly this point: by limiting the total reward allocable to each prompt, the advantage gap shrinks as correct samples increase, thereby suppressing excessively large update steps at the source without relying on post-hoc clipping or entropy regularization.

Limitations and Future Directions

The authors explicitly acknowledge that the current theoretical results mainly target the unclipped estimator, and equivalence is not strictly established for the clipped implementation actually used. Furthermore, the experimental scale is limited to two medium-sized models, and generalization to larger models and more diverse tasks still needs verification. However, these limitations do not diminish the core contribution—it identifies a neglected adjustment lever and provides theoretically supported empirical evidence.

Overall, the reward budgeting mechanism provides a simple and effective path for reinforcement learning training of LLM reasoning, and the update dynamics perspective behind it may also inspire further research on RL training stability.

One-line Summary

Reward budgeting constructs diminishing-return advantages by fixing the total reward per prompt, suppressing policy sharpening and improving LLM reasoning RL training.

Paragraph Summary

This article introduces a new study addressing premature convergence in reinforcement learning for LLM reasoning training. The authors point out that policy distributions become sharp during training, causing performance stagnation, and that merely increasing entropy cannot reliably break through the plateau, indicating that entropy is a symptom rather than the cause. Starting from update dynamics, the study derives a sharpness factor κ that describes the strength of reward gap-to-probability ratio gap conversion, and proposes the "reward budgeting" mechanism: assigning a fixed total reward to each prompt and constructing the advantage function under the budget constraint, so that returns diminish and the advantage gap shrinks as correct samples increase. On Qwen3-8B and Llama3.2-3B-Instruct, the method consistently improves multiple RL baselines, with up to +12.50 on AMC23 and +10.00 on AIME24/25, without explicit entropy regularization. The authors also acknowledge that the theory mainly targets the unclipped estimator, and equivalence for the clipped implementation has not been strictly proven.