VASPO: Reshaping Self-Distilled Reasoning Training with Privileged Value Estimation

Title

VASPO: Reshaping Self-Distilled Reasoning Training with Privileged Value Estimation

Meta Description

VASPO proposes fusing privileged value estimation with self-distillation signals within the PPO framework, complementarily integrating the two signals by analyzing their positional dynamics. On six mathematical reasoning benchmarks, Qwen3-4B and OLMo3-7B improve by 2.59 and 1.81 percentage points respectively. The code is open-sourced.

Article Content

Reasoning model training has long faced a contradiction: reinforcement learning signals are sparse, while self-distillation signals are too local. VASPO proposes a new scheme that fuses the strengths of both within the PPO framework, surpassing existing methods on multiple mathematical reasoning benchmarks.

The Problem: What Exactly Is Missing from Self-Distillation's Supervisory Signal

On-policy self-distillation's core idea is to have a "privileged self-teacher" provide dense supervision to the student model. Compared to sparse rewards that only look at whether the final answer is right or wrong, this approach is clearly more informative—every step of generation has gradients that can be backpropagated.

But the paper points out a key flaw: the self-teacher's supervision reflects local token preferences, not the expected return when the student model continues generating without guidance. In other words, the teacher tells you "this token is good," but doesn't say "can you ultimately reach the correct answer by continuing from here."

Some recent methods have attempted to combine self-distillation signals with GRPO: using GRPO to determine update direction and teacher signals for scaling. The authors of VASPO argue that this kind of integration does not fully realize the true potential of the two signals.

What is a privileged self-teacher? In the self-distillation paradigm, the same model is given two roles: one is the "student," which can only see the problem itself and generates step by step in the usual way; the other is the "teacher," which receives additional "privileged information" at each generation step, such as the standard answer, intermediate steps of the reference answer, or richer context. Based on this extra information, the teacher gives a probability distribution or preference score for each token, and the student is trained to match the teacher's distribution. Because the teacher's information is more abundant, the supervision it provides is far denser than the sparse reward of "only looking at final correctness," with a backpropagatable gradient signal at every step. This is also why on-policy self-distillation has attracted attention in reasoning tasks in recent years.

The relationship between sparse rewards and GRPO / PPO. In mathematical reasoning tasks, verifiable rewards are usually given only at the end of the sequence—correct answers get points, wrong ones do not—and the intermediate reasoning steps have no direct signal, which is what "reward sparsity" means. PPO (Proximal Policy Optimization) estimates the value of each state through a critic network and uses the advantage function to propagate the sparse reward at the end back to intermediate tokens, alleviating the sparsity problem. GRPO (Group Relative Policy Optimization) removes the independent critic and instead samples a group of responses to the same question, using relative scores within the group as the advantage baseline, thereby reducing memory and training costs. Both types of methods rely only on whether the final answer is correct, lacking fine-grained judgment about "whether a particular intermediate step is actually good," which is exactly the gap that self-distillation signals aim to fill.

Complementary Positional Dynamics: A Neglected Pattern in Training Signals

VASPO's core insight comes from observing how the two signals behave at different positions in the generated sequence.

The accuracy of privileged information for the critic's value prediction is stronger at the end of the response—the closer to the answer, the better the critic can judge whether the current state is good or bad. The self-teacher's guidance signal, however, shows a completely different pattern: strong at the beginning, decaying in the middle, and rising again at the end.

This complementarity means that if the right signal is used at the right position, training efficiency can be significantly improved. The combination of GRPO + teacher scaling precisely ignores this positional dynamic and simply superimposes the two signals together.

The meaning of "advantage," "value," and "token-level PPO advantage." In reinforcement learning, the value function V(s) estimates the expected cumulative return obtainable starting from state s, and the critic is the network that learns this function. The advantage A(s, a) = Q(s, a) − V(s) measures "how much better taking action a in state s is than average." In a language model, a token is an action, so token-level advantage measures whether generating this token at this position is good or bad relative to expectation. In VASPO, the critic uses predicted continuation return to assign an advantage score to each token, equivalent to providing "directional guidance" from a global perspective for the entire reasoning chain; the self-teacher signal, meanwhile, is a local preference judgment for a single token. The two operate over different scopes, which is the fundamental reason they can complement each other.

How VASPO Works: Value-Augmented Self-Distillation Policy Optimization

VASPO stands for Value-Augmented Self-Distillation Policy Optimization. Its architecture runs within the PPO framework and contains two key components that evaluate the same privileged information on the same prefix generated by the student:

The critic computes the token-level PPO advantage. It evaluates each token's contribution based on predicted continuation return, providing a value estimate from a global perspective.

The self-teacher modulates selected positive advantages. Through bounded probability ratios, the teacher signal fine-tunes the advantage values to avoid excessive intervention.

The cleverness of this design lies in its clear division of labor: the critic handles "is the direction right," while the teacher handles "is this step good," and the two play to their respective strengths at different positions.

Experimental Results: Stable Performance Improvements

The paper evaluates on six mathematical reasoning benchmarks. On Qwen3-4B, VASPO improves average accuracy by 2.59 percentage points over PPO; on OLMo3-7B, it improves by 1.81 percentage points.

On the Knights and Knaves task, VASPO achieves the highest average accuracy among all evaluated methods. Interestingly, these improvements are consistently observed across different model scales and architectures, indicating that the method itself generalizes well.

The code is open-sourced on GitHub (https://github.com/cii030/VASPO), facilitating reproduction and further research.

Why This Direction Deserves Attention

The value of VASPO lies not only in the numerical improvements, but also in the methodological question it reveals: simply combining two training signals is inferior to understanding their respective dynamic characteristics during training and then performing targeted integration.

This approach of "first analyzing signal characteristics, then designing a fusion strategy" has broad reference value for the growing number of signal sources in current reasoning model training (RL rewards, process reward models, self-distillation, contrastive learning, etc.). How to make different signals take effect at the right time and the right position may be more important than simply increasing the variety of signals.

Why bounded probability ratios matter. In PPO-style algorithms, the importance sampling ratio r = π_new(a|s) / π_old(a|s) is used to correct the difference between the new and old policy distributions, but if this ratio is unbounded, it may deviate sharply in a single update step, leading to unstable training or even collapse. The classic PPO approach uses clipping to constrain the ratio within the interval [1−ε, 1+ε]. VASPO lets the self-teacher signal modulate the advantage value through "bounded probability ratios," which essentially adds a safety ceiling to teacher intervention: the teacher can amplify or reduce a positive advantage, but the magnitude is limited to a controllable range, preserving fine-grained guidance while avoiding excessive teacher intervention in the update direction.

One-line Summary

VASPO dynamically fuses critic and self-distillation signals by position within the PPO framework, consistently surpassing PPO on reasoning benchmarks.

Paragraph Summary

VASPO (Value-Augmented Self-Distillation Policy Optimization) addresses the problems of sparse reinforcement learning rewards and overly limited self-distillation signals in reasoning model training by proposing a new scheme that fuses the two types of signals within the PPO framework. The paper's core insight is that privileged information makes the critic's value prediction more accurate at the end of the response, while the self-teacher's guidance signal shows positional dynamics of strong at the beginning, decaying in the middle, and rising again at the end—the two are exactly complementary. VASPO has the critic compute token-level PPO advantages to grasp the global direction, while the self-teacher modulates selected positive advantages through bounded probability ratios, achieving fine-grained local guidance. On six mathematical reasoning benchmarks, VASPO improves over PPO by an average of 2.59 percentage points on Qwen3-4B and 1.81 percentage points on OLMo3-7B, and achieves the highest average accuracy on the Knights and Knaves task. The code is open-sourced.