Reward Budget
A plug-in method for reinforcement learning training of large language models. It constructs the advantage function by assigning a fixed total reward budget to each prompt, producing diminishing returns and mitigating premature sharpening of the policy.
Timeline (last 90 days)
The reward budget mechanism allocates a fixed total reward budget to each prompt and constructs the advantage function under that budget constraint.
This theoretical result is intended to motivate the clipped implementation rather than to establish an exact objective equivalence.
On the two models Qwen3-8B and Llama3.2-3B-Instruct, the reward budget mechanism consistently improved multiple RL baselines.
On the AMC23 benchmark, the reward budget mechanism achieved a maximum improvement of +12.50.
On the AIME24 and AIME25 benchmarks, the reward budget mechanism achieved improvements of +10.00 each.
These gains were achieved without explicitly preserving entropy, directly challenging the common assumption that entropy must be maintained to avoid premature convergence.
By limiting the total reward for each prompt, the reward budget naturally produces diminishing returns, so that the policy does not become overly sharpened due to a few high-reward samples.
The current theoretical results mainly target the unclipped estimator; for the clipped implementation actually used, equivalence has not been strictly established.
The experimental scale is limited to two medium-sized models, and generalization capability on larger models and more diverse tasks still needs verification.
All Facts (9)
The reward budget mechanism allocates a fixed total reward budget to each prompt and constructs the advantage function under that budget constraint.
50%UnverifiedThis theoretical result is intended to motivate the clipped implementation rather than to establish an exact objective equivalence.
50%UnverifiedOn the two models Qwen3-8B and Llama3.2-3B-Instruct, the reward budget mechanism consistently improved multiple RL baselines.
50%UnverifiedOn the AMC23 benchmark, the reward budget mechanism achieved a maximum improvement of +12.50.
50%UnverifiedOn the AIME24 and AIME25 benchmarks, the reward budget mechanism achieved improvements of +10.00 each.
50%UnverifiedThese gains were achieved without explicitly preserving entropy, directly challenging the common assumption that entropy must be maintained to avoid premature convergence.
50%UnverifiedBy limiting the total reward for each prompt, the reward budget naturally produces diminishing returns, so that the policy does not become overly sharpened due to a few high-reward samples.
50%UnverifiedThe current theoretical results mainly target the unclipped estimator; for the clipped implementation actually used, equivalence has not been strictly established.
50%UnverifiedThe experimental scale is limited to two medium-sized models, and generalization capability on larger models and more diverse tasks still needs verification.
50%