APSD: Asymmetric Distillation by Correctness for 2.6x Faster LLM Post-Training

Title

APSD: Asymmetric Distillation by Correctness for 2.6x Faster LLM Post-Training

Meta Description

APSD proposes a correctness-dependent asymmetric self-distillation objective: correct rollouts use interpolation to preserve successful behavior, while incorrect rollouts use extrapolation to strengthen correction, achieving SDPO-level performance on Qwen3 models with 2.6x fewer training steps.

Article Content

Self-Distillation Policy Optimization (SDPO) provides dense token-level supervision signals for LLM post-training, with the core idea of using a feedback-conditioned "self-teacher" to guide the student model. However, directly matching the self-teacher's distribution creates a subtle tension: it may disrupt existing successful behavior on correct rollouts, while providing insufficient corrective force on incorrect rollouts. To address this, researchers propose APSD (Asymmetric Policy Optimization via Self-Distillation), which constructs a correctness-dependent asymmetric target distribution and adopts different mixing strategies in the two scenarios, thereby balancing behavior preservation and error correction.

The Pain Point of Self-Distillation: The Cost of One-Size-Fits-All Matching

SDPO's intuition is straightforward: let the model learn from its own "teacher version" given feedback. But the paper points out a key flaw—the self-teacher is not always superior to the student. In rollouts where the model has already generated correct answers, the self-teacher's distribution may deviate from the student's successful behavior, and forced alignment can weaken those already-correct generation patterns. In incorrect rollouts, merely matching the self-teacher may also be insufficiently aggressive, failing to provide gradient signals strong enough to correct errors.

This asymmetric phenomenon of "over-intervention when correct, under-correction when wrong" stems from the fact that the quality of the self-teacher distribution itself depends on the feedback condition, and the relative superiority of teacher versus student is inconsistent across different correctness states. APSD's starting point is precisely to acknowledge this asymmetry and design the target distribution accordingly.

Self-Distillation is a form of knowledge distillation that requires no external teacher model; the student model updates its parameters by matching distributions it produces itself under specific conditions (such as a feedback-augmented version). In LLM post-training, it is typically used to convert sparse sequence-level rewards into dense token-level supervision signals: given a question and feedback, the model generates a rollout, then uses the distribution from "re-evaluation under the feedback condition" as the teacher distribution, computing divergence loss token by token. Its advantage is that it requires no additional annotations or larger models, but the key assumption is that "the feedback-conditioned self-teacher is superior to the unconditional student"—and this assumption does not hold symmetrically for correct versus incorrect samples.

APSD's Core Mechanism: Interpolation and Extrapolation

APSD constructs a correctness-dependent target distribution by mixing between the student distribution and the self-teacher distribution, but the direction of mixing varies with correctness.

For correct rollouts, APSD adopts an interpolation strategy: the target distribution lies between the student and the self-teacher, preserving the student's already-successful generation behavior with only mild adjustment. This effectively "anchors" the student distribution on correct samples, preventing noise from the self-teacher from disrupting existing capabilities.

For incorrect rollouts, APSD adopts an extrapolation strategy: the target distribution extends beyond the self-teacher, strengthening the corrective force. This effectively "gives a push" on incorrect samples, making the model more decisively move away from erroneous patterns.

This asymmetric design directly addresses SDPO's pain point: conservative when correct, aggressive when wrong. The paper validates the design's rationale through three theoretical tools—divergence decomposition, reverse-KL barycenter geometry, and gradient analysis—with particular emphasis on the importance of gradient flow passing through the student-dependent target: the target distribution cannot be static but must dynamically adjust with the student distribution to ensure effective gradient propagation.

Interpolation and extrapolation are two common mixing directions used when constructing target distributions. Let the student distribution be (p_S) and the self-teacher distribution be (p_T). An interpolation target can be written as ((1-\alpha)p_S+\alpha p_T) ((0<\alpha<1)), with the target falling between the two, serving as a smoothing, conservative update; an extrapolation target takes the form (p_S+\beta(p_T-p_S)) ((\beta>1)), extending along the "student→teacher" direction past the teacher, thereby amplifying the teacher's corrective signal relative to the student. Interpolation naturally has a trust-region effect and is suitable for preserving existing correct behavior; extrapolation may introduce greater variance and needs to be used with clipping or KL constraints. APSD leverages this geometric difference to switch the mixing direction based on rollout correctness.

Reverse KL divergence refers to (\mathrm{KL}(p_T|p_S)), i.e., the divergence weighted by the teacher distribution and taken as expectation over the student distribution. Compared with forward KL, reverse KL has a "mode-seeking" property: when the teacher distribution is multimodal, the student is pulled toward one of the teacher's dominant modes rather than covering all modes. In self-distillation, the target distribution is often matched to the student in reverse-KL form, which shifts the gradient barycenter toward the teacher's high-probability regions, but also means the target distribution must change dynamically with the student, otherwise gradient flow degenerates. APSD's gradient analysis is precisely based on this property, showing that a student-dependent target distribution can provide more effective correction directions.

Experimental Validation: Strong Baseline Comparison and 2.6x Speedup

The paper conducts experiments in reinforcement learning settings with verifiable rewards and in rich feedback scenarios. APSD outperforms RL baselines and self-distillation baselines in overall performance, and on the Qwen3 model series, it achieves performance comparable to SDPO with an average of 2.6x fewer training steps.

This speedup ratio is noteworthy. In LLM post-training, training steps are directly tied to computational cost and iteration cycles. A 2.6x reduction in steps means APSD significantly improves sample efficiency while maintaining final performance. Combined with its asymmetric target design, it can be inferred that the speedup comes from more effective gradient signals: avoiding unnecessary perturbation on correct samples reduces wasted updates, while stronger correction on incorrect samples accelerates convergence.

From a methodological perspective, APSD's contribution lies in translating the observation that "self-teacher quality varies with correctness" into an actionable target distribution design. It introduces no additional teacher models or complex reward shaping, but rather performs asymmetric mixing within the self-distillation framework, preserving the method's simplicity.

Implications for LLM Post-Training

APSD's proposal reflects a broader trend: post-training optimization is moving from coarse-grained policy gradients toward fine-grained token-level supervision, while conditioning the quality of supervision signals. Self-distillation provides dense signals, but the "direction" and "strength" of the signals need to adapt based on sample correctness.

For practitioners, APSD's approach can be transferred to other feedback-driven training scenarios: when feedback signals vary in quality, one-size-fits-all distillation or alignment may backfire, and adjusting the target distribution conditionally is a more robust choice. The divergence decomposition and gradient analysis framework used in the paper also provides analytical tools for designing similar methods.

However, APSD's evaluation is currently concentrated on verifiable reward tasks and Qwen3 models, and its generalizability to more open scenarios with noisier feedback remains to be verified. In addition, correctness judgment itself relies on reliable verifiers, and how verifier errors affect the asymmetric strategy's effectiveness is a question worth attention in future work.

One-line Summary

APSD uses a correctness-dependent asymmetric self-distillation objective—interpolation to preserve behavior on correct rollouts and extrapolation to strengthen correction on incorrect rollouts—achieving 2.6x training speedup.

Paragraph Summary

This article introduces APSD (Asymmetric Policy Optimization via Self-Distillation), a self-distillation policy optimization method for LLM post-training. The article points out that SDPO's direct matching of the self-teacher distribution suffers from the asymmetric pain point of "over-intervention when correct, under-correction when wrong," because self-teacher quality varies with rollout correctness. APSD accordingly constructs a correctness-dependent target distribution: for correct rollouts it uses interpolation, anchoring the target between the student and self-teacher to preserve successful behavior; for incorrect rollouts it uses extrapolation, extending the target beyond the self-teacher to strengthen correction. The paper justifies the design through divergence decomposition, reverse-KL barycenter geometry, and gradient analysis, and experiments in verifiable-reward RL and rich-feedback scenarios show that APSD outperforms RL and self-distillation baselines, achieving SDPO-comparable performance on the Qwen3 series with an average of 2.6x fewer training steps, while keeping the method simple and introducing no additional teacher models.