VASPO
A reasoning model training method that combines reinforcement learning with self-distillation signals. Within the PPO framework, it uses privileged value estimates to enhance token-level advantages: the critic provides the global value direction, while the self-teacher regulates local positive advantages, improving training efficiency on mathematical reasoning tasks.
Timeline (last 90 days)
VASPO stands for Value-Augmented Self-Distillation Policy Optimization, and its architecture operates within the PPO framework, fusing self-distillation signals with PPO value estimation.
VASPO improves average accuracy by 2.59 percentage points over PPO on Qwen3-4B and by 1.81 percentage points on OLMo3-7B.
The paper evaluated VASPO on six mathematical reasoning benchmarks and achieved the highest average accuracy among all evaluated methods on the Knights and Knaves task.
VASPO contains two key components: the critic computes token-level PPO advantages based on predicted continuation returns, and the self-teacher modulates selected positive advantages through a bounded probability ratio.
The code for VASPO has been open-sourced on GitHub at https://github.com/cii030/VASPO.
The authors of VASPO believe that simply combining self-distillation signals with GRPO (using GRPO to determine the update direction and using teacher signals for scaling) does not realize the true potential of the two signals.
The improvements of VASPO are consistently observed across different model scales and architectures, indicating that the method has good generalization.
All Facts (7)
VASPO contains two key components: the critic computes token-level PPO advantages based on predicted continuation returns, and the self-teacher modulates selected positive advantages through a bounded probability ratio.
50%UnverifiedVASPO improves average accuracy by 2.59 percentage points over PPO on Qwen3-4B and by 1.81 percentage points on OLMo3-7B.
50%UnverifiedThe paper evaluated VASPO on six mathematical reasoning benchmarks and achieved the highest average accuracy among all evaluated methods on the Knights and Knaves task.
50%UnverifiedVASPO stands for Value-Augmented Self-Distillation Policy Optimization, and its architecture operates within the PPO framework, fusing self-distillation signals with PPO value estimation.
50%UnverifiedThe improvements of VASPO are consistently observed across different model scales and architectures, indicating that the method has good generalization.
50%UnverifiedThe authors of VASPO believe that simply combining self-distillation signals with GRPO (using GRPO to determine the update direction and using teacher signals for scaling) does not realize the true potential of the two signals.
50%UnverifiedThe code for VASPO has been open-sourced on GitHub at https://github.com/cii030/VASPO.
50%