On-policy distillation
A knowledge distillation paradigm in which the student model receives token-level supervision from the teacher on trajectories it samples in real time under its current parameters. In contrast to offline distillation, which learns from fixed sequences, it aims to align the training and inference distributions.
Timeline (last 90 days)
The training data for on-policy distillation is generated by real-time sampling of the student model under its current parameters, rather than being a pre-prepared fixed corpus, which contrasts with offline distillation.
The paper finds that when the student's trajectory moves in the wrong direction, the teacher does not correct the error but instead continues writing along the student's error, that is, the teacher has the ability to recognize errors but lacks the ability to escape from them.
The authors suggest: do not assume by default that the teacher is reliable under any prefix; one should first test the teacher's performance on the student's erroneous trajectories, and if it only agrees, then the supervision signal itself is noise.
All Facts (3)
The training data for on-policy distillation is generated by real-time sampling of the student model under its current parameters, rather than being a pre-prepared fixed corpus, which contrasts with offline distillation.
50%UnverifiedThe paper finds that when the student's trajectory moves in the wrong direction, the teacher does not correct the error but instead continues writing along the student's error, that is, the teacher has the ability to recognize errors but lacks the ability to escape from them.
50%UnverifiedThe authors suggest: do not assume by default that the teacher is reliable under any prefix; one should first test the teacher's performance on the student's erroneous trajectories, and if it only agrees, then the supervision signal itself is noise.
50%