Exposure Bias
A problem in sequence generation models in which training uses ground-truth prefixes while inference uses prefixes generated by the model itself, leading to distribution inconsistency and error accumulation.
Timeline (last 90 days)
Traditional knowledge distillation uses an off-policy approach, having the student imitate the teacher's output distribution on a fixed dataset, which leads to exposure bias.
In on-policy distillation, the student model generates its own rankings during training and the teacher provides feedback on these rankings, thereby avoiding the mismatch between teacher and student distributions (exposure bias) in offline distillation.
Autoregressive models suffer from the exposure bias problem, whereby the model sees real prefixes during training but faces self-generated erroneous prefixes during inference, causing errors to accumulate progressively.
All Facts (3)
Traditional knowledge distillation uses an off-policy approach, having the student imitate the teacher's output distribution on a fixed dataset, which leads to exposure bias.
50%UnverifiedIn on-policy distillation, the student model generates its own rankings during training and the teacher provides feedback on these rankings, thereby avoiding the mismatch between teacher and student distributions (exposure bias) in offline distillation.
50%UnverifiedAutoregressive models suffer from the exposure bias problem, whereby the model sees real prefixes during training but faces self-generated erroneous prefixes during inference, causing errors to accumulate progressively.
50%