[KongchangAI]
· 2 min read· 1,140 words

MCPO: Allocating Token Credit on Demand for More Efficient Reinforcement Learning

MCPO: Allocating Token Credit on Demand for More Efficient Reinforcement Learning

MCPO uses top-2 margin to allocate RL credit to hesitant tokens, stably beating GRPO across models, though the longer-answer confound remains.

To address GRPO's flaw of distributing sequence-level rewards evenly across all tokens, MCPO measures each token's hesitation via the top-2 margin (difference between highest and second-highest token probabilities) and maps it to scores through fixed thresholds, allocating more credit to uncertain positions while keeping the mean total advantage unchanged. Read directly from the trainer's existing forward pass, it requires no extra evaluation or batch statistics, keeping implementation costs low. On MiniCPM5-2B-Base and Qwen3-4B-Base, MCPO stably outperforms the GRPO baseline across multiple random seeds, with macro-average avg@32 gains of 6.55 and 4.75 points respectively, and up to 13.8 points at pass@32. However, MCPO-trained models produce answers 2 to 3 times longer than the baseline, and the experiments failed to separate the length effect from the credit allocation effect—an open limitation worth clarifying in future research.

In the combination of reinforcement learning with verifiable rewards (RLVR), algorithms such as GRPO typically distribute the correctness reward of the final answer evenly across all generated tokens. But a paper from RSS proposes a different approach: allocate credit only where the model hesitates, and based on this designed the MCPO (Margin-Credit Policy Optimization) method.

MCPO paper source screenshot

The Core Problem: GRPO's "Egalitarian" Flaw

GRPO works in a straightforward way: the model generates an answer, and if it is ultimately correct, a reward is given and passed equally to every token of that answer. This is simple to implement but ignores the differences among tokens in the sequence—some tokens the model is very confident about, while others are in a state of hesitation.

The authors of MCPO argue that credit should be allocated more to the tokens where the model hesitates, because these positions are the key nodes where the model learns new knowledge.

GRPO (Group Relative Policy Optimization) is a reinforcement learning optimization algorithm proposed by the DeepSeekAd team, whose core idea is to perform relative comparisons within a group of sampled responses rather than relying on an independent value network to estimate the baseline. Specifically, for the same question, the model generates multiple responses, computes the reward for each response, then uses the group's average reward as the baseline, subtracts this baseline from each response's reward to obtain the advantage value, and uses this advantage value to update the policy. This design eliminates the cost of training a value model, making the reinforcement learning pipeline more concise, and is therefore widely adopted in RLVR (reinforcement learning with verifiable rewards) scenarios. The key to understanding GRPO is that the advantage value is a scalar applied to the entire sequence, and how to distribute this sequence-level signal to individual tokens is precisely the aspect MCPO attempts to improve.

How MCPO Measures "Hesitation"

MCPO uses the top-2 margin of the policy's next-token distribution to measure the degree of hesitation: ( M = p_1 - p_2 ), i.e., the probability difference between the highest-probability token and the second-highest. The smaller the margin, the more uncertain the model is between two candidate tokens.

Then, two fixed thresholds map the margin to a score, and while keeping the mean total advantage of each solution unchanged, the advantage is dispersed across tokens. This means the average level of the overall reward signal remains unchanged, but the allocation weights change.

This method has a practical advantage: the margin is read directly from the forward pass already available in the trainer, requiring no additional model evaluation and no batch statistics, so the implementation cost is very low.

Experimental Setup and Main Results

The authors used LoRA to train MiniCPM5-2B-Base for 200 steps on a dataset of 2,048 DAPO-Math problems, and evaluated on six competition math benchmarks.

  • Compared to the unmodified GRPO baseline, MCPO achieved improvements on all five data random seeds, with a macro-average avg@32 improvement of 6.55 points on average (95% confidence interval [4.1, 9.0]).
  • Reproduction experiments on Qwen3-4B-Base also showed improvements, with all three seeds positive and an average improvement of 4.75 points.

Where Do the Improvements Come From?

Analysis shows that the improvements are most pronounced on problems that "GRPO can only occasionally solve," and they expand as k in pass@k increases, reaching a 13.8-point improvement at pass@32. This indicates that MCPO helps the model improve coverage on problems that were originally unstable.

An Unsolved Mystery: Longer Answers

The answers written by the MCPO-trained model are 2 to 3 times longer than the baseline, and runs that write longer answers typically score higher. However, the authors explicitly state that the experiments failed to separate these two effects: that is, it cannot be determined whether the performance improvement comes from longer answers or from the margin-guided credit allocation itself. This is an important open limitation.

In reinforcement learning training, the relationship between answer length and performance is a classic confounding factor. Existing research has shown that certain policy optimization methods inadvertently encourage models to generate longer outputs, because longer sequences may obtain higher cumulative signals under token-level average rewards, or because long answers themselves contain more reasoning steps and are thus more likely to be correct. To separate the length effect from the credit allocation effect, controlled variable experiments are usually needed, such as penalizing or truncating length, comparing under the same length budget, or using length-normalized rewards. The authors acknowledge that the experiments failed to do this, which means that part of MCPO's gain over GRPO may come merely from the side effect of "writing longer" rather than from margin-guided credit allocation itself. This limitation poses a substantial challenge to the causal interpretation of the conclusions and is a priority issue for subsequent reproduction and extension research.

Multi-Source Perspectives and Significance

According to the paper abstract from this RSS source, the core innovation of MCPO lies in aligning token-level credit allocation with model uncertainty. Compared to other methods that require additional value networks or complex statistics, MCPO's lightweight design makes it easy to integrate into existing training pipelines.

However, since there is only one source (the paper itself) and no independent reproduction, the confounding factor of "longer answers" still needs to be clarified by future research.

Conclusion

MCPO provides a simple yet effective idea: using the model's own hesitation signals to reallocate credit in reinforcement learning. The stable improvements across multiple models and seeds suggest that this token-level credit allocation strategy is worth further exploration. But the confounding factor of answer length also reminds us to remain cautious when interpreting the results.

One-line Summary

MCPO uses the top-2 margin to allocate reinforcement learning credit to tokens where the model hesitates, stably outperforming GRPO across multiple models, but the confounding factor of longer answers has not yet been ruled out.

Paragraph Summary

To address GRPO's "egalitarian" flaw of distributing sequence-level rewards evenly across all tokens, MCPO proposes using the top-2 margin (the difference between the highest and second-highest token probabilities) to measure each token's degree of hesitation, and maps the margin to a score via fixed thresholds, allocating more credit to positions where the model is uncertain while keeping the mean total advantage unchanged. The method reads the margin directly from the forward pass already available in the trainer, requiring no additional evaluation or batch statistics, resulting in low implementation cost. On MiniCPM5-2B-Base and Qwen3-4B-Base, MCPO stably outperforms the GRPO baseline across multiple random seeds, with macro-average avg@32 improvements of 6.55 and 4.75 points respectively, and an improvement of up to 13.8 points at pass@32. However, the answers generated by the MCPO-trained model are 2 to 3 times longer than the baseline, and the experiments failed to separate the length effect from the credit allocation effect—an open limitation worth clarifying in future research.

#MCPO#GRPO#reinforcement learning#RLVR#token credit allocation#top-2 margin
Share:

Related articles