MCPO
A reinforcement learning policy optimization method that assigns credit by measuring the model's uncertainty at each token (the top-2 margin), concentrating advantage more heavily on tokens where the model is hesitant in order to improve learning efficiency.
Timeline (last 90 days)
MCPO (Margin-Credit Policy Optimization) is a reinforcement learning method based on token-level credit assignment, whose core idea is to allocate more credit to tokens where the model hesitates.
MCPO uses the top-2 margin (the probability difference between the highest-probability and second-highest-probability tokens) of the policy's next-token distribution to measure the model's degree of hesitation, with a smaller margin indicating greater model uncertainty.
MCPO maps the margin to a score via two fixed thresholds and, while keeping the mean total advantage of each solution unchanged, distributes the advantage across tokens.
MCPO's margin can be read directly from the forward pass already available in the trainer, requiring no additional model evaluation or batch statistics, so the implementation cost is very low.
Compared with the unmodified GRPO baseline, MCPO achieves improvements on all five data random seeds, with the macro-average avg@32 improving by 6.55 points on average (95% confidence interval [4.1, 9.0]).
Reproduction experiments on Qwen3-4B-Base show that MCPO also improves over GRPO, with all three seeds positive and an average improvement of 4.75 points.
MCPO's improvement is most pronounced on problems that 'GRPO can only occasionally solve', and it grows as k in pass@k increases, reaching a 13.8-point improvement at pass@32.
Answers written by the MCPO-trained model are 2 to 3 times the length of the baseline, and runs that write longer typically score higher.
The authors explicitly state that the experiments failed to separate the two effects of answer length and credit assignment, so it cannot be determined whether the performance gain comes from longer answers or from margin-guided credit assignment itself.
The MCPO paper comes from an RSS source, and currently there is only one source (the paper itself), lacking independent reproduction.
All Facts (10)
MCPO (Margin-Credit Policy Optimization) is a reinforcement learning method based on token-level credit assignment, whose core idea is to allocate more credit to tokens where the model hesitates.
50%UnverifiedMCPO uses the top-2 margin (the probability difference between the highest-probability and second-highest-probability tokens) of the policy's next-token distribution to measure the model's degree of hesitation, with a smaller margin indicating greater model uncertainty.
50%UnverifiedMCPO maps the margin to a score via two fixed thresholds and, while keeping the mean total advantage of each solution unchanged, distributes the advantage across tokens.
50%UnverifiedMCPO's margin can be read directly from the forward pass already available in the trainer, requiring no additional model evaluation or batch statistics, so the implementation cost is very low.
50%UnverifiedCompared with the unmodified GRPO baseline, MCPO achieves improvements on all five data random seeds, with the macro-average avg@32 improving by 6.55 points on average (95% confidence interval [4.1, 9.0]).
50%UnverifiedReproduction experiments on Qwen3-4B-Base show that MCPO also improves over GRPO, with all three seeds positive and an average improvement of 4.75 points.
50%UnverifiedMCPO's improvement is most pronounced on problems that 'GRPO can only occasionally solve', and it grows as k in pass@k increases, reaching a 13.8-point improvement at pass@32.
50%UnverifiedAnswers written by the MCPO-trained model are 2 to 3 times the length of the baseline, and runs that write longer typically score higher.
50%UnverifiedThe authors explicitly state that the experiments failed to separate the two effects of answer length and credit assignment, so it cannot be determined whether the performance gain comes from longer answers or from margin-guided credit assignment itself.
50%UnverifiedThe MCPO paper comes from an RSS source, and currently there is only one source (the paper itself), lacking independent reproduction.
50%