APSD
A self-distillation-based post-training method for large models that applies asymmetric target distributions to correct and incorrect rollouts, interpolating for correct ones and extrapolating for incorrect ones, so as to balance behavior preservation and error correction.
Timeline (last 90 days)
The researchers propose APSD (Asymmetric Policy Optimization via Self-Distillation), which balances behavior preservation and error correction by constructing a correctness-dependent asymmetric target distribution.
For correct rollouts, APSD adopts an interpolation strategy, with the target distribution lying between the student and the self-teacher, preserving the student's already-successful generation behaviors.
For incorrect rollouts, APSD adopts an extrapolation strategy, extending the target distribution beyond the self-teacher to strengthen correction.
APSD's experimental results show that it outperforms reinforcement learning baselines and self-distillation baselines in overall performance.
Across the Qwen3 model scale series, APSD reaches performance comparable to SDPO with an average of 2.6 times fewer training steps.
APSD's target distribution must be adjusted dynamically with the student distribution to ensure effective gradient propagation.
APSD's evaluation is currently concentrated on verifiable-reward tasks and Qwen3 models, and its generalization to more open scenarios with noisier feedback remains to be verified.
All Facts (7)
The researchers propose APSD (Asymmetric Policy Optimization via Self-Distillation), which balances behavior preservation and error correction by constructing a correctness-dependent asymmetric target distribution.
50%UnverifiedFor correct rollouts, APSD adopts an interpolation strategy, with the target distribution lying between the student and the self-teacher, preserving the student's already-successful generation behaviors.
50%UnverifiedFor incorrect rollouts, APSD adopts an extrapolation strategy, extending the target distribution beyond the self-teacher to strengthen correction.
50%UnverifiedAPSD's experimental results show that it outperforms reinforcement learning baselines and self-distillation baselines in overall performance.
50%UnverifiedAcross the Qwen3 model scale series, APSD reaches performance comparable to SDPO with an average of 2.6 times fewer training steps.
50%UnverifiedAPSD's target distribution must be adjusted dynamically with the student distribution to ensure effective gradient propagation.
50%UnverifiedAPSD's evaluation is currently concentrated on verifiable-reward tasks and Qwen3 models, and its generalization to more open scenarios with noisier feedback remains to be verified.
50%