Waypoint-OPD: Migrating RL Policy Shifts via Gated Distillation

Reinforcement Learning (RL) training can significantly boost large models' performance on tasks like mathematical reasoning and code generation, but this improvement is hard to transfer directly to other models. One approach is distillation—extracting the teacher model's policy change before and after RL training and using it as a supervision signal to teach the student model. However, this "endpoint-style" transfer has a fundamental blind spot: looking only at the policy difference between start and end points makes it impossible to tell whether that change has been consistent.

The Blind Spot of Endpoint Distillation: Same Shift, Different Paths

Existing methods use the difference in the teacher model's log probabilities before and after RL as a supervision signal, applied to trajectories generated by the student model itself. This is essentially telling the student: "The teacher became more/less favorable toward this action."

But the problem is that the teacher's policy shift is a dynamic process. Early training may strongly favor a certain action direction, while later stages reverse or even partially cancel it out. If we only look at endpoints, we cannot distinguish between "a consistently positive shift" and "a sharp early rise followed by a late decline"—their endpoint differences may be identical.

This missing information leads to reduced transfer efficiency: the student may learn a preference that the teacher itself never stably held.

Policy distillation is a general paradigm for transferring a teacher model's behavior to a student model, typically achieved by minimizing the divergence (e.g., KL divergence) between the student's output distribution and the teacher's output distribution. In the RL post-training setting, the teacher model's output distribution changes continuously during training, so the distillation supervision signal becomes "the teacher policy's relative change" rather than the teacher's absolute output. Here, the "log probability difference" refers to the difference in the teacher model's log probability for a given action before and after RL, often used as a proxy for the advantage signal: a large difference means the action was reinforced more strongly by RL, while a small or negative difference indicates it was suppressed. It is important to distinguish that endpoint-style distillation uses only the initial and final checkpoints, which is a "finite difference" approximation; policy shifts during training may be non-monotonic, and endpoint differences cannot capture the stability of their sign—this is precisely the blind spot Waypoint-OPD aims to address.

Waypoint Guidance: Using Intermediate Checkpoints to Determine Shift Direction

The core idea of Waypoint-OPD is to introduce an intermediate checkpoint as a "waypoint" to verify the directional consistency of the teacher's policy shift.

Specifically, researchers take an intermediate checkpoint from the same RL training run and, at prefix positions visited by the student model, compute the relative action preference changes for two phases: early (initial→intermediate) and late (intermediate→endpoint). If the early and late shifts point in the same direction, the teacher's preference evolution is consistent; if they point in opposite directions, the endpoint shift may be misleading.

Based on this directional consistency, the method gates the endpoint shift: it is retained when directions agree and suppressed when they conflict. This way, the supervision signal received by the student more faithfully reflects the policy improvements the teacher stably holds.

In terms of implementation cost, Waypoint-OPD requires only one additional frozen-teacher scoring forward pass, making the computational overhead very small.

A checkpoint is a snapshot of model parameters saved at fixed step or time intervals during training. RL training typically saves one checkpoint per evaluation cycle, so "intermediate checkpoints" are naturally available without extra training cost. In Waypoint-OPD, the intermediate checkpoint plays the role of a "waypoint," analogous to using sampled points to approximate a function curve in numerical analysis. A "prefix position" refers to an intermediate state the student model reaches while generating its own trajectory; only by computing teacher shifts at these positions can the supervision signal be guaranteed to lie on the state distribution the student actually visits, avoiding distribution mismatch. "Gating" is a common mechanism in deep learning that uses a scalar or vector to weight or scale another signal path; here the gate value is derived from directional consistency, approaching 1 when directions agree and 0 when they conflict, thereby preserving effective supervision while suppressing misleading signals.

Experimental Results: Leading in All Six Settings

Researchers validated the method on two domains: mathematical reasoning and code reasoning, using RL-trained Qwen3-4B as the teacher model and Qwen3-8B, Qwen3-14B, and Qwen3-30B-A3B as student models.

Across all six "domain × student model" combinations, Waypoint-OPD achieved the highest domain average scores among the compared methods. This means the gains from directional consistency gating are not limited to a specific model scale or task type.

Ablation experiments further confirmed the necessity of the gating mechanism: with Qwen3-8B as the student, removing the gate or randomly shuffling it led to a drop in mathematical reasoning performance. This indicates that the directional consistency judgment itself carries effective signal, rather than merely adding noise.

Domain average score refers to the average score across multiple evaluation benchmarks within a task domain (such as mathematical reasoning or code reasoning), used to reduce the chance effects of fluctuations in a single benchmark. An ablation study removes or replaces a component of the model and observes changes in performance to verify that component's necessity; here, replacing the gate with random values (randomly shuffling the gate) is meant to control for the possible alternative explanation that "it merely adds a multiplicative perturbation." If a random gate also achieved the same gains, it would suggest the directional consistency judgment has no real effect; conversely, it supports the authors' causal explanation. It is worth noting that Qwen3-4B was used as the teacher while three different scales of Qwen3 served as students, meaning the experiments controlled along the dimension of "fixed teacher, varying student scale," but the range of teacher model scales itself has not yet been covered.

Consistency or Not: A Quality Criterion for Distillation Signals

This work offers a noteworthy perspective: in policy distillation, the temporal structure of the supervision signal may be as important as the signal itself.

Traditional distillation methods treat the teacher as a static black box, caring only about the input-output mapping. Waypoint-OPD instead leverages trajectory information naturally present during RL training—the evolution direction of the same action preference at different training stages—to assess which signals are worth transmitting.

The limitations of this approach are also obvious: it depends on being able to access intermediate checkpoints from teacher training, so it does not apply to scenarios where only the final model is available. Moreover, the influence of the intermediate checkpoint selection strategy (which time point to choose as the waypoint) on final performance has not been fully explored.

Nevertheless, as RL training increasingly becomes standard for large-model post-training, Waypoint-OPD provides an illuminating answer to how to more effectively use the intermediate information generated during training.

The core concept of this work can be understood within the more general framework of the temporal structure of supervision signals. Traditional supervised learning assumes labels are time-independent, but in settings such as RL post-training and continual learning, supervision signals drift with the training process, and the signal's "path" may actually be more informative than its "endpoint value." Related ideas include curriculum learning (organizing sample order by difficulty or stage) and GAE in advantage estimation (generalized advantage estimation, using multi-step differences to reduce the variance of single-step estimates). Waypoint-OPD can be seen as introducing the idea of "multi-step differencing" into cross-model distillation, using intermediate checkpoints to test the stability of the shift direction. Its limitations are also clear: it requires access to intermediate checkpoints from teacher training and does not apply to scenarios where only the final model weights are open; at the same time, the choice of waypoint (early, middle, or near the endpoint) is essentially a trade-off between information content and representativeness, with no theoretical guidance yet.

One-line Summary

Waypoint-OPD uses intermediate checkpoints to determine whether the teacher's policy shift direction is consistent, gating the distillation signal, and leads in all six settings.

Paragraph Summary

RL can improve large models' reasoning ability, but the teacher model's policy shift before and after RL is hard to transfer directly to the student model. Existing endpoint distillation only compares log probability differences between initial and final checkpoints, making it impossible to distinguish between "a consistently positive shift" and "a sharp early rise followed by a late decline"—two very different evolutionary paths. Waypoint-OPD introduces an intermediate checkpoint as a waypoint and, at prefix positions visited by the student, separately computes the relative action preference direction for early and late phases; when directions agree, it retains the endpoint shift, and when they conflict, it suppresses it. The method requires only one additional frozen-teacher scoring forward pass. With Qwen3-4B as the teacher and three different scales of student models, it achieved the highest domain average scores in all six settings across mathematical and code reasoning; ablation experiments show that removing or shuffling the gate degrades mathematical reasoning performance.