On-Policy Distillation
A distillation technique in large model training in which the student model continuously generates responses using its current policy during training, and a teacher model scores those responses or provides probability-based supervision, enabling the student to learn from and correct its current errors.
Core Facts
Timeline (last 90 days)
OPD does not require task success/failure reward signals and relies only on the teacher's token-by-token supervision, so it is a reward-free method.
The researchers found that even starting from the states where the student deviates the most, the teacher's own takeover can still complete the task, whereas the student, after training on the teacher's per-token scores, often runs out of turns.
The core challenge of On-Policy Distillation (OPD) is how to extract effective supervisory signals from the teacher model; traditional methods rely on token probabilities or directly aligning hidden states, each with its own limitations: token probabilities may underestimate the teacher's internal knowledge, while directly aligning hidden states forces accommodation to the teacher's feature space, which is especially tricky in cross-architecture transfer.
Token-level on-policy distillation aligns the student model's hidden states with teacher signals and adds a gating mechanism for credit assignment, yielding improvements on math and tool-use benchmarks.
The key mechanism of RM-OPD is applying token-level dense supervision on critiques generated by the student itself, making the supervision distribution consistent with the student's current policy distribution and alleviating the training-inference distribution mismatch problem.
The arXiv paper "RM-OPD: On-Policy Distillation for Generative Reward Models" proposes a two-stage process: first train multiple domain-specific reward model teachers, then integrate them into one generative student model through routed on-policy distillation, and this unified student model is said to surpass all of its distilled expert teachers.
On-policy distillation (OPD) has the student model generate its own trajectories, and then the teacher model provides supervision signals on those trajectories.
Version 2.0 switched to online policy distillation, where the model generates its own responses and a teacher model scores each token; two teachers were set up — the original model plus hidden human instructions for chat, and a pure base model for instruction following, tool calling, and coding.
Online policy distillation (OPD) has the student model generate its own training responses, which are then supervised by the teacher model.
All Facts (9)
Online policy distillation (OPD) has the student model generate its own training responses, which are then supervised by the teacher model.
75%UnverifiedThe researchers found that even starting from the states where the student deviates the most, the teacher's own takeover can still complete the task, whereas the student, after training on the teacher's per-token scores, often runs out of turns.
50%UnverifiedOPD does not require task success/failure reward signals and relies only on the teacher's token-by-token supervision, so it is a reward-free method.
50%UnverifiedThe core challenge of On-Policy Distillation (OPD) is how to extract effective supervisory signals from the teacher model; traditional methods rely on token probabilities or directly aligning hidden states, each with its own limitations: token probabilities may underestimate the teacher's internal knowledge, while directly aligning hidden states forces accommodation to the teacher's feature space, which is especially tricky in cross-architecture transfer.
50%UnverifiedToken-level on-policy distillation aligns the student model's hidden states with teacher signals and adds a gating mechanism for credit assignment, yielding improvements on math and tool-use benchmarks.
50%UnverifiedThe arXiv paper "RM-OPD: On-Policy Distillation for Generative Reward Models" proposes a two-stage process: first train multiple domain-specific reward model teachers, then integrate them into one generative student model through routed on-policy distillation, and this unified student model is said to surpass all of its distilled expert teachers.
50%UnverifiedThe key mechanism of RM-OPD is applying token-level dense supervision on critiques generated by the student itself, making the supervision distribution consistent with the student's current policy distribution and alleviating the training-inference distribution mismatch problem.
50%UnverifiedOn-policy distillation (OPD) has the student model generate its own trajectories, and then the teacher model provides supervision signals on those trajectories.
50%UnverifiedVersion 2.0 switched to online policy distillation, where the model generates its own responses and a teacher model scores each token; two teachers were set up — the original model plus hidden human instructions for chat, and a pure base model for instruction following, tool calling, and coding.
50%