OC-OPD
An on-policy knowledge distillation method that coordinates two complementary supervision signals, forward KL and reverse KL, through a four-observation matrix. By employing a Fisher-inverse metric for coupling, it realizes a closed-form negotiated gradient update. The approach balances the mode-covering behavior of forward KL divergence against the mode-seeking behavior of reverse KL divergence within a unified distillation objective.
Timeline (last 90 days)
OC-OPD is an on-policy distillation method that uses a four-observation matrix to coordinate the gradients of forward KL and reverse KL, avoiding manual weight search.
OC-OPD constructs a four-observation matrix: treating the student and teacher as candidate source and evaluator respectively, self-observations (student evaluating student, teacher evaluating teacher) measure uncertainty, and cross-observations correspond to the reverse-KL and forward-KL objectives respectively.
The paper establishes an exact identity: coupling the Fisher-inverse metrics of the two KL logit gradients is equivalent to the sum of these contrastive observations.
OC-OPD's update procedure has three steps: observations set the preference, closed-form negotiation coordinates the forward and reverse KL gradients, and the distribution discrepancy scales the update magnitude.
The paper evaluates OC-OPD under three student settings (non-thinking generation, thinking generation, and base-model initialization), achieving clear gains on mathematical reasoning tasks and transferring to out-of-distribution science QA and code generation tasks.
OC-OPD's code is publicly available on GitHub at https://github.com/RuoranXu/OC-OPD.
OC-OPD replaces part of the manual hyperparameter tuning with observation-driven preferences and closed-form negotiation, which may be one reason it works under different student settings.
All Facts (7)
OC-OPD is an on-policy distillation method that uses a four-observation matrix to coordinate the gradients of forward KL and reverse KL, avoiding manual weight search.
50%UnverifiedOC-OPD constructs a four-observation matrix: treating the student and teacher as candidate source and evaluator respectively, self-observations (student evaluating student, teacher evaluating teacher) measure uncertainty, and cross-observations correspond to the reverse-KL and forward-KL objectives respectively.
50%UnverifiedThe paper establishes an exact identity: coupling the Fisher-inverse metrics of the two KL logit gradients is equivalent to the sum of these contrastive observations.
50%UnverifiedOC-OPD's update procedure has three steps: observations set the preference, closed-form negotiation coordinates the forward and reverse KL gradients, and the distribution discrepancy scales the update magnitude.
50%UnverifiedThe paper evaluates OC-OPD under three student settings (non-thinking generation, thinking generation, and base-model initialization), achieving clear gains on mathematical reasoning tasks and transferring to out-of-distribution science QA and code generation tasks.
50%UnverifiedOC-OPD's code is publicly available on GitHub at https://github.com/RuoranXu/OC-OPD.
50%UnverifiedOC-OPD replaces part of the manual hyperparameter tuning with observation-driven preferences and closed-form negotiation, which may be one reason it works under different student settings.
50%