RL for Reasoning Only Changes 1-3% of Tokens? The Truth and Controversy Behind the Claimed 1000x Compute Savings

An Underappreciated Key Insight

Recently, a paper sparking widespread discussion put forward a rather disruptive conclusion: reinforcement learning (RL), when used to improve the reasoning capabilities of large models, actually only changes 1-3% of the output tokens. Even more strikingly, the researchers claim they can reproduce the same performance gains without using RL, at approximately 1000x lower computational cost.

If this conclusion holds, it would have profound implications for the current mainstream training paradigm for reasoning models. From OpenAI's o1 series to DeepSeek-R1, the industry has widely regarded RL as the core mechanism for unlocking a model's "deep thinking" capabilities, investing massive compute resources toward this end. This paper suggests that perhaps we've been using a cannon to kill a mosquito all along.

The Central Role of RL in Reasoning Model Training

To understand the disruptive nature of this paper, we first need to understand the role RL plays in current reasoning model training. The application of reinforcement learning in large language models primarily refers to the third stage after pretraining and supervised fine-tuning—using reward signals to guide the model in optimizing its output strategy. Starting with OpenAI's InstructGPT, RLHF (Reinforcement Learning from Human Feedback) became the standard process for aligning models with human preferences. In the domain of reasoning models, RL's role has been further amplified: by designing verifiable reward functions based on mathematical correctness, logical consistency, and other criteria, models learn to generate longer, more structured Chains-of-Thought (CoT). Representative examples of this paradigm include OpenAI's o1 series (using large-scale RL training to improve mathematical and coding reasoning) and DeepSeek-R1 (where reasoning behavior emerges from a base model through pure RL). The core algorithms for RL training are typically based on PPO (Proximal Policy Optimization) or its variant GRPO, with computational overhead far exceeding that of supervised learning due to the need for repeated sampling, evaluation, and policy updates. It is precisely this steep computational cost that makes the claim of "1000x compute savings" so attention-grabbing.

What Does a 1-3% Token Difference Mean?

In the Reddit discussion, one commenter hit the nail on the head: "This insight alone is worth being the core of a paper, and shouldn't be brushed aside as a premise in the methodology section."

This remark highlights a common problem in current AI research—truly valuable findings are often buried beneath engineering-focused narratives. If the model only differs in 1-3% of output tokens before and after RL training, it reveals several deep questions:

Reasoning Ability May Already "Exist" in the Base Model

If RL only fine-tunes an extremely small portion of tokens, then the base model itself likely already possesses the knowledge and capability needed for reasoning. RL's role is more about "activating" or "guiding" the expression of this capability rather than building reasoning pathways from scratch. This aligns with recent research perspectives suggesting that "reasoning ability is awakened rather than created."

Here we need to understand the essential characteristics of base models. A base model refers to a raw language model that has undergone large-scale corpus pretraining but has not yet been instruction-tuned or aligned. During the pretraining phase, through the next-token prediction task, the model implicitly acquires language structure, world knowledge, and even a degree of logical reasoning patterns from trillions of tokens of text. Recent research shows that many capabilities attributed to post-training stages actually already exist in a "dormant" state during the pretraining phase. For example, Anthropic's research has found that base models can demonstrate complex reasoning behavior under appropriate prompt guidance; and DeepSeek-R1's paper partially supports this view—they discovered that even without SFT cold-start, pure RL can "activate" reasoning capabilities from a base model, implying that these capabilities are already encoded in the model weights. If a 1-3% token change can release these latent capabilities, then our narrative about RL "teaching" models to reason needs fundamental revision.

Enormous Room for Training Efficiency Optimization

Since the core changes are so sparse, using lighter methods—such as supervised fine-tuning (SFT), rejection sampling, or even carefully designed prompt engineering—to approximate RL's effects becomes theoretically feasible. The 1000x compute savings claimed by the paper is built precisely on this logic.

Specifically, rejection sampling (also called Best-of-N sampling) is a relatively lightweight strategy optimization method: for a given problem, the model generates multiple candidate answers, and then a verifier or reward model selects the best answer as training data. This approach avoids the complex policy gradient calculations and value function estimation in RL, essentially transforming the RL problem into a supervised learning problem. DeepSeek-R1's training pipeline includes extensive rejection sampling-based SFT stages to distill the high-quality reasoning patterns discovered during the RL phase into stable training signals. If RL indeed only changes 1-3% of token outputs, then using rejection sampling to identify the differential patterns of these key tokens, followed by direct injection via SFT, could theoretically approximate RL's effects at extremely low computational cost. This is the technical foundation for the paper's claimed 1000x compute savings—replacing RL's "carpet bombing" with precise "surgical operations."

Why Do "Small Changes" Lead to "Major Degradation"?

However, another sharper challenge in the discussion deserves the attention of all practitioners:

"1-3% of tokens sounds small, but it shouldn't translate to a 30% drop in actual usability."

This touches on the chasm between statistical metrics and real-world experience. The commenter astutely categorized it as a pervasive phenomenon: the credibility trap of claims like "our model is 10x smaller but only 2% worse."

Non-Uniform Distribution of Token Differences

The crux of the issue is that the 1-3% figure is an average that obscures the distributional structure of differences. In reasoning tasks, certain critical tokens (such as logical turning points, key computation steps, and conclusion judgments) carry far more weight than ordinary connectors or modifiers. If this 1-3% happens to be concentrated at these "critical junctures," then the collapse of overall output quality becomes entirely understandable.

From an information-theoretic perspective, the contribution of tokens at different positions to overall semantics is highly non-uniform. Key decision tokens (such as "therefore" and "assume" in mathematical reasoning, or conditional judgment words in logical branches) carry far higher information entropy than filler tokens (such as "the," "this," "then"). In a reasoning chain, an error in one critical step contaminates all subsequent reasoning through cascade effects (error propagation), while changes in connectors typically don't affect the final conclusion. This phenomenon is similar to the "critical path" concept in software engineering—even if the code accounts for only 5%, code on the critical path determines the correctness of the entire system. Therefore, analyzing the specific positions and semantic roles of the 1-3% of tokens changed by RL is far more meaningful than focusing on the percentage itself.

In other words, which tokens change matters more than how many tokens change. A paper that only emphasizes "just 1-3%" without analyzing the semantic importance of these tokens risks having its reproduction claims overestimated.

The Gap Between Benchmarks and Real-World Usability

This also explains why many "small models approaching large models" studies perform impressively on benchmarks but reveal shortcomings once deployed in practice. Benchmarks typically only examine the correctness of final answers and cannot adequately capture the robustness of reasoning chains, error recovery capability, and handling quality of edge cases. A 2% benchmark gap may be amplified into a 30% decline in user experience in real application scenarios.

Commonly used benchmarks in AI such as MATH, GSM8K, AIME, and HumanEval typically use final answer accuracy (Pass@1 or Pass@k) as their core metric. This evaluation approach has systematic blind spots: it cannot distinguish between "arriving at the correct answer by chance" and "reaching the correct answer through robust reasoning." A model might perform perfectly on 80% of common problem types but frequently collapse in complex scenarios involving multi-step reasoning that requires error detection and backtracking. Furthermore, benchmarks typically have fixed distributional characteristics, and models can achieve inflated scores by "memorizing" similar problems from training data (i.e., data contamination). This is why a 2% benchmark gap may correspond to a 30% decline in actual experience—real-world usage scenarios have far more long-tail and diverse distributions than benchmarks, with higher demands on reasoning chain robustness.

Reflections on the AI Research Paradigm

The value of this discussion perhaps transcends the technical details of the paper itself. It reminds us to maintain triple clarity when evaluating AI progress:

First, be wary of the deceptiveness of aggregate metrics. Whether it's token change rates or benchmark scores, aggregated numbers can mask critical structural differences. Truly effective evaluation should deeply analyze "where exactly the differences occur."

Second, reproduction claims require more rigorous verification. Claiming to "achieve equivalent results with 1000x less compute" is an extremely strong assertion. If true, the massive compute the industry has invested in RL would face reexamination; but until independently reproduced and validated at scale, this conclusion should be treated with caution.

Third, genuine insights deserve to be highlighted. As the original poster advocated, if the finding that "RL only changes 1-3% of tokens" is real, it is in itself an important clue for understanding how reasoning models work and deserves to be studied in depth as an independent proposition, rather than serving merely as a prelude to cost-reduction methods.

Conclusion

This paper and the ensuing discussion are fundamentally asking a root question: Do we truly understand how reinforcement learning improves the reasoning capabilities of large models? If the answer is "it merely adjusts a sparse set of key tokens," then the entire landscape of reasoning model training could face restructuring.

But on the other side, the chasm between 1-3% token differences and 30% usability degradation also warns us: in the AI field, simple numbers often hide complex truths. For any method claiming to be "cheaper, faster, and nearly lossless," the most rational attitude remains—stay curious, but wait for more substantial evidence.