KL Distillation
A knowledge distillation method in which the original full-attention model serves as the teacher and the student model imitates the teacher's output distribution, with the KL divergence as the loss.
Timeline (last 90 days)
ZHA requires only a brief KL distillation training at the scale of a few hundred billion tokens to be applied to existing pretrained models, without pretraining from scratch.
KL distillation uses the original full-attention model as the teacher and has the student model, with its structure replaced by ZHA, imitate the teacher's next-token distribution, with KL divergence as the loss.
All Facts (2)
ZHA requires only a brief KL distillation training at the scale of a few hundred billion tokens to be applied to existing pretrained models, without pretraining from scratch.
50%UnverifiedKL distillation uses the original full-attention model as the teacher and has the student model, with its structure replaced by ZHA, imitate the teacher's next-token distribution, with KL divergence as the loss.
50%