KL Self-Distillation
A variant of knowledge distillation in which the teacher model and student model share the same set of parameters or architecture; the student learns by minimizing the KL divergence between its output distribution and that of the teacher, yielding smoother output distributions.
Core Facts
Timeline (last 90 days)
The gradient of standard MTP contains two components: a KL self-distillation signal and a noise residual.
The KL self-distillation signal helps improve the acceptance rate of speculative decoding.
Soft-MTP decouples the KL self-distillation signal from the noise residual, so that during gradient backpropagation the gradient flows only through the KL self-distillation part.
KL divergence is used to compare the difference in output probability distributions between the fine-tuned model and the original model given the same input; a larger value indicates greater deviation, and a KLD of 0.0244 means the fine-tuning has hardly changed the model's overall behavioral distribution.
All Facts (4)
KL divergence is used to compare the difference in output probability distributions between the fine-tuned model and the original model given the same input; a larger value indicates greater deviation, and a KLD of 0.0244 means the fine-tuning has hardly changed the model's overall behavioral distribution.
65%UnverifiedThe gradient of standard MTP contains two components: a KL self-distillation signal and a noise residual.
50%UnverifiedThe KL self-distillation signal helps improve the acceptance rate of speculative decoding.
50%UnverifiedSoft-MTP decouples the KL self-distillation signal from the noise residual, so that during gradient backpropagation the gradient flows only through the KL self-distillation part.
50%