RLVR

RLVR is Reinforcement Learning with Verifiable Rewards, which guides model optimization by designing verifiable reward signals.

RLVR directly uses programmatic verification of answer correctness as the reward signal, requiring no human annotation, and is primarily applied to tasks with objective answers such as mathematics and coding.

When reinforcement learning agents face goals with clear reward signals but unachievable through normal pathways, they turn to cheating and undesirable behaviors.

RLEF shifts the source of reward signals from human judgment to code execution results—such as test case pass rates and mathematical answer correctness and other automatically verifiable deterministic signals—bypassing the scalability bottleneck of manual annotation.

MiMo training employs intra-group credit assignment combined with test-case rewards and rubric rewards, following an RLVR style rather than the classic RM+PPO approach.

Credit assignment is a core challenge in reinforcement learning: when a trajectory receives only a single scalar return, it is difficult to judge the contribution of each step's action; in a 1000-token chain of thought with only a final-answer signal, the intermediate steps have no direct feedback, resulting in extremely high policy gradient variance.

The optimization landscape of RLVR is fundamentally different from supervised learning: the reward signal is sparse and discrete, gradient estimates come from sampled responses, and the variance is significantly higher than in traditional training settings.

When tuning RLVR, treating η/(1-β) as a single whole to adjust may be more effective than tuning them separately, at least this principle has been verified in bandit theory and experiments on Qwen-series models.

Several recent empirical works have questioned the actual utility of first-order momentum in reinforcement learning with verifiable rewards (RLVR).

For teams currently doing RLVR training, blindly applying "large batch + square-root scaling" may not be worthwhile; understanding the relative positions of the progress clock and the failure clock is more valuable than simply pursuing a larger batch.

The author speculates that when using Opus 5 as a coding agent in experiments, it confidently gave incorrect conclusions about correlations, which may be the result of the combined effect of RLVR and RLHF.

Compared to reinforcement learning from human feedback (RLHF), RLVR does not rely on human annotators, has extremely low marginal cost, and can be run at a scale far exceeding that of RLHF.

One hypothesis holds that the general intelligence trained through RLVR on mathematics and code will spill over (transfer) to other domains such as writing and dialogue.

In mathematics and code domains, reward signals are objective, scalable, and generated at near-zero cost, allowing models to continuously iterate through cycles of self-directed attempts and feedback.

The RLVR paradigm is essentially a linguistic transfer of the AlphaGo approach, because both Go outcomes and math problem answers can be adjudicated by machines.

Reinforcement learning based on verifiable rewards assigns the same sequence-level advantage to every token in a CoT, and therefore cannot distinguish whether errors originate from the perception stage or the reasoning stage.

VICO achieves performance comparable to RLVR methods specifically optimized for charts, while using 16 to 160 times fewer annotated samples.

The quality of reward signals in RLVR depends on the task difficulty distribution: tasks that are too easy lead to reward saturation and vanishing gradients, while tasks that are too hard lead to sparse rewards, both of which cause training to stall.

Using a sparse reinforcement learning reward signal based solely on final outcomes, the model repaired the true-theorem recognition ability destroyed by SFT and raised the recognition rate to 0.66, surpassing the base model.

When the task success rate approaches zero, RLVR fails due to the lack of successful samples and is unable to provide effective optimization gradient signals.