[KongchangAI]
ModelStochastic Horizons On-Policy Distillation

SH-OPD

An improved on-policy distillation method that reduces redundant supervision and improves computational efficiency by randomly sampling a shared horizon upper bound at the batch level.

Timeline (last 90 days)

Nov 17

Stochastic Horizons On-Policy Distillation (SH-OPD) is an improved on-policy distillation method whose core idea is to randomly vary the horizon length of rollouts, reducing supervision of redundant positions while retaining access to deep response states.

Unverified50%
Nov 17

SH-OPD samples one shared horizon cap per training batch rather than scheduling each sequence individually, thereby enabling efficient parallel generation and avoiding the overhead of per-sequence horizon scheduling.

Unverified50%
Nov 17

Random horizons introduce a reweighting bias: tokens have different probabilities of being included in supervision under different horizons, and simple averaging causes the gradient estimate to deviate from the goal of full-trajectory supervision; the authors characterize this bias and derive the trade-off between compute and reweighting.

Unverified50%
Nov 17

On mathematical reasoning tasks, SH-OPD achieves mathematical reasoning accuracy slightly higher than the baseline at about 45% of the single-update cost.

Unverified50%
Nov 17

On code generation tasks, SH-OPD maintains competitive code generation performance at about 60% of the baseline cost.

Unverified50%
Nov 17

Allocating the supervision budget more intelligently across response depth is more cost-effective than simply lengthening trajectories, and this conclusion has direct engineering value for on-policy training scenarios that require large numbers of rollouts.

Unverified50%
Nov 17

The effectiveness of SH-OPD has so far been validated mainly on mathematical reasoning and code generation tasks, and transfer to more open-domain tasks still requires further observation.

Unverified50%

All Facts (7)

Unverified

Stochastic Horizons On-Policy Distillation (SH-OPD) is an improved on-policy distillation method whose core idea is to randomly vary the horizon length of rollouts, reducing supervision of redundant positions while retaining access to deep response states.

50%
Unverified

SH-OPD samples one shared horizon cap per training batch rather than scheduling each sequence individually, thereby enabling efficient parallel generation and avoiding the overhead of per-sequence horizon scheduling.

50%
Unverified

Random horizons introduce a reweighting bias: tokens have different probabilities of being included in supervision under different horizons, and simple averaging causes the gradient estimate to deviate from the goal of full-trajectory supervision; the authors characterize this bias and derive the trade-off between compute and reweighting.

50%
Unverified

On mathematical reasoning tasks, SH-OPD achieves mathematical reasoning accuracy slightly higher than the baseline at about 45% of the single-update cost.

50%
Unverified

On code generation tasks, SH-OPD maintains competitive code generation performance at about 60% of the baseline cost.

50%
Unverified

Allocating the supervision budget more intelligently across response depth is more cost-effective than simply lengthening trajectories, and this conclusion has direct engineering value for on-policy training scenarios that require large numbers of rollouts.

50%
Unverified

The effectiveness of SH-OPD has so far been validated mainly on mathematical reasoning and code generation tasks, and transfer to more open-domain tasks still requires further observation.

50%

Source Articles