SH-OPD
An improved on-policy distillation method that reduces redundant supervision and improves computational efficiency by randomly sampling a shared horizon upper bound at the batch level.
Timeline (last 90 days)
Stochastic Horizons On-Policy Distillation (SH-OPD) is an improved on-policy distillation method whose core idea is to randomly vary the horizon length of rollouts, reducing supervision of redundant positions while retaining access to deep response states.
SH-OPD samples one shared horizon cap per training batch rather than scheduling each sequence individually, thereby enabling efficient parallel generation and avoiding the overhead of per-sequence horizon scheduling.
Random horizons introduce a reweighting bias: tokens have different probabilities of being included in supervision under different horizons, and simple averaging causes the gradient estimate to deviate from the goal of full-trajectory supervision; the authors characterize this bias and derive the trade-off between compute and reweighting.
On mathematical reasoning tasks, SH-OPD achieves mathematical reasoning accuracy slightly higher than the baseline at about 45% of the single-update cost.
On code generation tasks, SH-OPD maintains competitive code generation performance at about 60% of the baseline cost.
Allocating the supervision budget more intelligently across response depth is more cost-effective than simply lengthening trajectories, and this conclusion has direct engineering value for on-policy training scenarios that require large numbers of rollouts.
The effectiveness of SH-OPD has so far been validated mainly on mathematical reasoning and code generation tasks, and transfer to more open-domain tasks still requires further observation.
All Facts (7)
Stochastic Horizons On-Policy Distillation (SH-OPD) is an improved on-policy distillation method whose core idea is to randomly vary the horizon length of rollouts, reducing supervision of redundant positions while retaining access to deep response states.
50%UnverifiedSH-OPD samples one shared horizon cap per training batch rather than scheduling each sequence individually, thereby enabling efficient parallel generation and avoiding the overhead of per-sequence horizon scheduling.
50%UnverifiedRandom horizons introduce a reweighting bias: tokens have different probabilities of being included in supervision under different horizons, and simple averaging causes the gradient estimate to deviate from the goal of full-trajectory supervision; the authors characterize this bias and derive the trade-off between compute and reweighting.
50%UnverifiedOn mathematical reasoning tasks, SH-OPD achieves mathematical reasoning accuracy slightly higher than the baseline at about 45% of the single-update cost.
50%UnverifiedOn code generation tasks, SH-OPD maintains competitive code generation performance at about 60% of the baseline cost.
50%UnverifiedAllocating the supervision budget more intelligently across response depth is more cost-effective than simply lengthening trajectories, and this conclusion has direct engineering value for on-policy training scenarios that require large numbers of rollouts.
50%UnverifiedThe effectiveness of SH-OPD has so far been validated mainly on mathematical reasoning and code generation tasks, and transfer to more open-domain tasks still requires further observation.
50%