ConceptDistillation Loss
Distillation Loss
In sequence generation tasks, a loss function computed as token-level cross-entropy over selected trajectories, used to make the student model maximize the likelihood of the trajectory sequence given the input.
Timeline (last 90 days)
Oct 19
Optimizing only the trace loss may cause the student to overfit to the surface form of the reasoning text while ignoring the correctness of the conclusion; optimizing only the answer loss degenerates into traditional answer distillation and loses the transfer of reasoning ability.
Unverified50%