CriticOPD
A conceptual framework that improves on-policy distillation training. The teacher first critiques the student's erroneous trajectories, the student rewrites accordingly, and the teacher then scores the revised output using the critique as privileged information. This internalizes error-correction ability within the inference phase without relying on external prompts.
Timeline (last 90 days)
The paper proposes the concept of teacher recoverability: given the state prefix of any student at any step, the teacher can follow that prefix to find a reasonable path to the correct answer.
The paper proposes the CriticOPD method, whose process has three steps: the teacher gives a critique of the student's erroneous trajectory, the student rewrites the solution process based on the critique, and the teacher scores the revised output with the critique provided to the teacher as privileged information.
CriticOPD adopts an asymmetric design: the teacher has the critique as additional information when scoring, while the critique is removed when training the student, so the student can only rely on the revised output and the teacher's scoring signal.
A critique is the teacher's explicit verbalized analysis of the error location and error cause, which switches the teacher from continuation mode to diagnostic mode.
The paper validated the effectiveness of CriticOPD on two Qwen3 student models and six mathematical reasoning benchmarks, and the results show that it brings stable improvements over OPD.
CriticOPD yields the largest gains on competition-style math problems, because competition problems have longer step chains, making it easier for the student to go astray midway and for the teacher to follow the error with higher probability.
CriticOPD only requires modifying the way supervision is generated in the training pipeline, without changing the model structure or the form of the loss function, and can be directly layered on top of existing online distillation pipelines.
The authors suggest: do not assume by default that the teacher is reliable under any prefix; one should first test the teacher's performance on the student's erroneous trajectories, and if it only agrees, then the supervision signal itself is noise.
All Facts (8)
The paper proposes the concept of teacher recoverability: given the state prefix of any student at any step, the teacher can follow that prefix to find a reasonable path to the correct answer.
50%UnverifiedThe paper proposes the CriticOPD method, whose process has three steps: the teacher gives a critique of the student's erroneous trajectory, the student rewrites the solution process based on the critique, and the teacher scores the revised output with the critique provided to the teacher as privileged information.
50%UnverifiedCriticOPD adopts an asymmetric design: the teacher has the critique as additional information when scoring, while the critique is removed when training the student, so the student can only rely on the revised output and the teacher's scoring signal.
50%UnverifiedA critique is the teacher's explicit verbalized analysis of the error location and error cause, which switches the teacher from continuation mode to diagnostic mode.
50%UnverifiedThe paper validated the effectiveness of CriticOPD on two Qwen3 student models and six mathematical reasoning benchmarks, and the results show that it brings stable improvements over OPD.
50%UnverifiedCriticOPD yields the largest gains on competition-style math problems, because competition problems have longer step chains, making it easier for the student to go astray midway and for the teacher to follow the error with higher probability.
50%UnverifiedCriticOPD only requires modifying the way supervision is generated in the training pipeline, without changing the model structure or the form of the loss function, and can be directly layered on top of existing online distillation pipelines.
50%UnverifiedThe authors suggest: do not assume by default that the teacher is reliable under any prefix; one should first test the teacher's performance on the student's erroneous trajectories, and if it only agrees, then the supervision signal itself is noise.
50%