The RLVR Generalization Dilemma: Why Reasoning Strategies Fail Under OOD

Reinforcement Learning with Verifiable Rewards (RLVR) is regarded as an important path to improving large language model reasoning, but a new study reveals a counterintuitive phenomenon: models perform excellently on in-distribution tasks yet collapse when transferred out-of-distribution (OOD). Through a controllable equation-solving framework, this paper analyzes in depth the causes of RLVR generalization failure and proposes an unexpected mitigation strategy.

Research Background: RLVR's Generalization Promise and Hidden Concerns

RLVR optimizes model outputs through verifiable reward signals (such as answer correctness) and has shown potential in tasks like mathematical reasoning and code generation. Its core assumption is that the model can not only learn to solve tasks within the training distribution but also generalize reasoning strategies to new scenarios.

However, the paper points out that this generalization is not a given. The authors designed a controllable equation-solving framework—given a non-bijective string function f, the model is required to find multiple inputs x such that f(x)=y. This setup makes reasoning strategies observable while supporting well-defined "easy-to-hard" OOD shifts (e.g., more solutions, longer y, etc.). Through this framework, researchers can precisely track changes in model behavior both in-distribution and out-of-distribution.

Bijective vs. Non-bijective Functions: A bijection refers to a one-to-one mapping, where each input corresponds to exactly one output and each output to exactly one input. In the equation-solving framework, a bijective function f means that given y there is a unique solution x, and the model needs to perform a deterministic inverse operation; a non-bijective function, by contrast, allows multiple x to map to the same y, and the model must search for multiple solutions. This structural difference directly affects the reasoning strategies the model adopts: bijective tasks naturally force precise inverse reasoning, while non-bijective tasks may induce enumeration or heuristic search. Understanding this distinction helps grasp why the paper chooses bijective inversion tasks as a co-training target and how they contrast with the final non-bijective OOD evaluation tasks.

Core Finding: Strategy Collapse of RLVR Under OOD

Experiments reveal a subtle RLVR failure mode: in-distribution, the model not only achieves high accuracy but also generates structured step-by-step derivations; but under OOD shifts, the same model abandons derivation and degenerates into inefficient brute-force enumeration, causing accuracy to plummet.

This phenomenon rules out two common explanations: first, insufficient capability of the base model—because it already exhibits correct reasoning strategies in-distribution; second, reward hacking—because the model genuinely succeeds on in-distribution tasks. The authors argue that this exposes a deeper challenge for RLVR generalization: the reasoning strategies the model learns do not truly generalize but collapse under out-of-distribution pressure.

Out-of-Distribution (OOD) vs. In-Distribution (ID): In machine learning, in-distribution means test data comes from the same distribution as training data, while out-of-distribution means test data comes from a different distribution (e.g., harder, longer, or structurally different tasks). The OOD shifts in this study are constructed in a controllable manner, such as increasing the number of solutions or lengthening the target string y, thereby systematically changing task difficulty and structure. This setup allows researchers to observe whether the model can still maintain its reasoning strategies in "familiar but slightly harder" scenarios. OOD generalization is a key metric for evaluating a model's true reasoning ability, because real-world tasks often deviate from the training distribution.

Unexpected Solution: Co-training with Bijective Inversion Tasks

The key to mitigating this failure is an indirect intervention: co-training with bijective inversion tasks. Bijective tasks (i.e., one-to-one functions) are not actually uniformly harder, and are even farther from the final non-bijective OOD evaluation tasks. Yet it is precisely these tasks that make the model's derivation more deterministic and significantly restore correct reasoning strategies and accuracy under OOD shifts.

This finding is surprising because it contradicts the intuition that "the closer to the evaluation task, the more helpful." The authors further explain through theoretical analysis: outcome-only RLVR can shape the reasoning process by implicitly giving more credit to effective reasoning strategies, even without process supervision.

Reward Hacking: Refers to the model exploiting loopholes or shortcuts in the reward function to obtain high rewards rather than truly learning the desired behavior. In RLVR, if the reward is based only on final answer correctness, the model may learn to guess or enumerate rather than perform effective reasoning. The paper rules out the reward hacking explanation through success on in-distribution tasks, because the model does generate structured step-by-step derivations in-distribution rather than cutting corners. This concept is crucial for understanding RLVR's failure modes, as it distinguishes between two different problems: "strategies not generalizing" and "strategy degeneration."

Theoretical Insights and Implications

The paper's theoretical analysis shows that outcome-only RLVR does not merely optimize the final answer; it can implicitly incentivize the model to adopt more effective reasoning strategies. The reason co-training on bijective tasks works may be that such tasks require the model to perform deterministic inverse reasoning, thereby reinforcing the determinism of derivation and making the model more inclined to persist in structured reasoning rather than degenerate into enumeration when facing OOD shifts.

This work systematically reveals the key role of reasoning strategies in RLVR generalization and provides an analyzable, intervenable research framework. The code is open-sourced at https://github.com/ziqipang/rl_meta_thinking, providing a foundation for future research.

Implicit Credit Assignment: In outcome-only reinforcement learning, the reward signal acts only on the final output, and the model must infer on its own which intermediate steps contributed to success. Implicit credit assignment refers to the model indirectly assigning higher credit to effective reasoning strategies by optimizing the final outcome, thereby reinforcing those strategies. The paper's theoretical analysis shows that even without process supervision, RLVR can shape the reasoning process through this mechanism. Co-training on bijective tasks may reinforce the model's credit assignment for structured reasoning by increasing samples of deterministic inverse reasoning, enabling it to persist in effective strategies under OOD pressure. This concept bridges outcome supervision and process supervision and explains the potential mechanism of RLVR generalization.

Implications for AI Reasoning Research

For researchers and developers focused on LLM reasoning capabilities, this study conveys several important signals: RLVR generalization cannot rely solely on expanding the training distribution; the stability of reasoning strategies requires dedicated design; indirect auxiliary tasks may be more effective than directly simulating OOD tasks; and the boundary between outcome supervision and process supervision may be blurrier than expected. In the future, how to design reward structures and training curricula that induce robust reasoning strategies will become a key issue for RLVR to move toward practical application.

One-line Summary

RLVR excels at in-distribution reasoning but collapses under OOD; co-training with bijective tasks can mitigate this.

Paragraph Summary

This study reveals a counterintuitive generalization failure when RLVR improves large model reasoning: models can generate structured derivations on in-distribution tasks, but under out-of-distribution shifts they abandon reasoning and degenerate into brute-force enumeration, causing accuracy to plummet. The failure is neither due to insufficient base capability nor reward hacking, but rather that reasoning strategies do not truly generalize. Unexpectedly, co-training with bijective inversion tasks significantly mitigates OOD collapse, even though these tasks are farther from the evaluation distribution. Theoretical analysis shows that outcome-only RLVR can shape reasoning strategies through implicit credit assignment. The research provides a controllable equation-solving framework and open-source code, emphasizing that the stability of reasoning strategies requires dedicated design and that indirect auxiliary tasks may be more effective than directly simulating OOD.