D²-OPD
An improved method for online policy distillation that strengthens the learning signal by explicitly decoupling the direction and distance of extrapolation. It comprises two components: a per-prefix hard KL constraint and adaptive direction combination.
Timeline (last 90 days)
D²-OPD explicitly decouples the direction and distance of extrapolation and achieves significant improvements under multiple settings.
D²-OPD introduces a per-prefix hard KL constraint that sets an explicit upper bound for each prefix, ensuring that the KL divergence between the extrapolated target distribution and the teacher distribution does not exceed a preset threshold.
D²-OPD determines the extrapolation direction by adaptively weighted combination of the teacher model's post-training offset and the initial student-teacher gap.
D²-OPD optimizes the student model using forward KL divergence, so as to also cover low-probability tokens in the teacher distribution.
D²-OPD achieves the highest scores under all three settings on math reasoning benchmarks, with an improvement of up to 9.45 points over the strongest baseline method.
Ablation experiments show that all three components of D²-OPD—the hard KL constraint, adaptive direction combination, and forward KL optimization—each contribute positively to final performance.
D²-OPD's per-prefix hard KL constraint has controllable computational overhead, and the adaptive direction combination requires no additional teacher forward passes, giving it good feasibility in practical training.
The current experiments of D²-OPD are mainly concentrated on math reasoning benchmarks, and its performance on other task types such as open-ended generation and multi-turn dialogue remains to be verified.
The D²-OPD paper does not discuss in detail the threshold selection strategy for the hard KL constraint, which may be a key hyperparameter affecting the method's practical deployment.
All Facts (9)
D²-OPD explicitly decouples the direction and distance of extrapolation and achieves significant improvements under multiple settings.
50%UnverifiedD²-OPD introduces a per-prefix hard KL constraint that sets an explicit upper bound for each prefix, ensuring that the KL divergence between the extrapolated target distribution and the teacher distribution does not exceed a preset threshold.
50%UnverifiedD²-OPD determines the extrapolation direction by adaptively weighted combination of the teacher model's post-training offset and the initial student-teacher gap.
50%UnverifiedD²-OPD optimizes the student model using forward KL divergence, so as to also cover low-probability tokens in the teacher distribution.
50%UnverifiedD²-OPD achieves the highest scores under all three settings on math reasoning benchmarks, with an improvement of up to 9.45 points over the strongest baseline method.
50%UnverifiedAblation experiments show that all three components of D²-OPD—the hard KL constraint, adaptive direction combination, and forward KL optimization—each contribute positively to final performance.
50%UnverifiedD²-OPD's per-prefix hard KL constraint has controllable computational overhead, and the adaptive direction combination requires no additional teacher forward passes, giving it good feasibility in practical training.
50%UnverifiedThe current experiments of D²-OPD are mainly concentrated on math reasoning benchmarks, and its performance on other task types such as open-ended generation and multi-turn dialogue remains to be verified.
50%UnverifiedThe D²-OPD paper does not discuss in detail the threshold selection strategy for the hard KL constraint, which may be a key hyperparameter affecting the method's practical deployment.
50%