Knowledge Distillation
A model compression technique that transfers knowledge from a large teacher model to a lightweight student model. Soft labels convey inter-class semantic structure. Methods are categorized into logits-based, feature-based, and relation-based approaches.
Core Facts
Knowledge distillation transfers the full probability distribution information output by the teacher model via soft labels, producing an implicit data augmentation effect.
Knowledge distillation was proposed by Hinton et al. in 2015, originally referring to training a small model using the soft label outputs of a large model.
Knowledge distillation was first proposed by Hinton et al. in 2015.
Timeline (last 90 days)
Distillation, quantization, and pruning are often collectively called the three-piece model compression toolkit, but distillation changes the training signal, while the latter two directly compress existing weights.
For teams that rely on distillation for model compression or acceleration, beyond task performance they also need to detect behavioral shifts in the student model.
A student model distilled from a given teacher model may inherit the teacher's safety stance, whether or not the developers are explicitly aware of it.
Distilling from a teacher that accepts harmful content reduces the student model's harm-avoidance ability, while distilling from a teacher that avoids harmful content improves it, and this shift is an independent behavioral transfer rather than a byproduct of task ability.
A study named the phenomenon in which a student model inherits the teacher model's behavioral tendencies on unrelated tasks "Imprinting," with the paper titled "Imprinting: Knowledge Distillation Can Shift Student Behavior."
The product of distillation is a specialized model that can be deployed locally with extremely low inference cost, often used for tasks with clear boundaries such as classification, extraction, and intent recognition.
The limitation of distillation is that the student model's capability ceiling is constrained by the teacher's performance on that task, and it is difficult to generalize to new tasks outside the training distribution.
Teacher-student distillation pipeline: use a cloud large model to label edge cases, then distill into a small on-device model, and manually review samples where the two disagree.
The key risk of knowledge distillation is that the teacher also makes mistakes, especially on long-tail categories, and its errors are inherited verbatim by the student.
Over the past two years, techniques such as model quantization, pruning, and knowledge distillation have allowed hundred-billion-parameter models to run on a single consumer-grade graphics card or even on a phone.
40 more timeline events
All Facts (20)
Knowledge distillation transfers the full probability distribution information output by the teacher model via soft labels, producing an implicit data augmentation effect.
90%VerifiedKnowledge distillation was proposed by Hinton et al. in 2015, originally referring to training a small model using the soft label outputs of a large model.
90%VerifiedKnowledge distillation was first proposed by Hinton et al. in 2015.
90%VerifiedKnowledge Distillation is a technique for transferring the knowledge of a large model to a smaller, lightweight model, capable of retaining most of the performance while reducing the parameter count by tens of times.
80%Verified知识蒸馏由Hinton等人在2015年提出，核心思想是让小型学生模型模仿大型教师模型的输出分布
80%Verified模型蒸馏是将大模型的能力压缩到小模型中以降低推理成本的技术
80%VerifiedThis experiment adopts trajectory-level distillation, which belongs to the category of behavioral cloning, directly having the student model imitate the external behavioral sequences of the teacher model.
75%VerifiedKnowledge distillation was first proposed by Hinton, Vinyals, and Dean at Google in 2015, with the core idea of having a small student model learn from the soft labels output by a large teacher model.
75%VerifiedDistilBERT reduces BERT's parameter count by 40% through distillation, increases inference speed by 60%, and incurs only approximately 3% performance loss
75%VerifiedKnowledge distillation was systematized by Hinton et al. in 2015; its core idea is to transfer the dark knowledge of a large model to a small model, using soft probability distributions as supervision signals.
75%VerifiedFlash series models acquire capabilities through knowledge distillation from Pro or larger-scale Ultra models.
75%VerifiedModel distillation transfers the knowledge of a large model to a smaller student model, resulting in significant speed improvements but with some capability loss.
75%VerifiedKnowledge distillation, proposed by Hinton et al., has student models inherit implicit knowledge by learning from the soft probability distributions output by teacher models.
70%VerifiedLightweight models are compressed in scale through knowledge distillation and model pruning techniques; knowledge distillation has the small model mimic the output probability distribution of the large model, while pruning removes network weights that contribute least to the output.
65%Verified模型压缩涵盖量化、剪枝、知识蒸馏三条主要技术路径
65%UnverifiedThrough distillation, student models with only one-tenth the parameters of the teacher model can often retain 80%–95% of the teacher model's performance.
70%UnverifiedThe course builds a character-level language model that learns one letter at a time
60%UnverifiedThe teacher model outputs in distillation training rarely include self-correction examples, causing student models to form an implicit belief that "strong models don't make mistakes," leading them to repeat rather than correct when verification fails.
60%UnverifiedDistillation, quantization, and pruning are often collectively called the three-piece model compression toolkit, but distillation changes the training signal, while the latter two directly compress existing weights.
50%UnverifiedA study named the phenomenon in which a student model inherits the teacher model's behavioral tendencies on unrelated tasks "Imprinting," with the paper titled "Imprinting: Knowledge Distillation Can Shift Student Behavior."
50%