Knowledge Distillation Explained: How Small Models Learn from Large Ones

What Is Knowledge Distillation

Knowledge Distillation is a key model compression technique in deep learning. The core idea is straightforward: use a large, capable Teacher Model to guide the training of a smaller, faster Student Model, allowing the smaller model to dramatically reduce computational and storage costs while retaining as much of the larger model's performance as possible.

Recently, a content creator shared a video on Reddit explaining the principles behind knowledge distillation, sparking community discussion around the topic. This article draws on that discussion to break down how knowledge distillation works and why it matters.

As large language models and neural networks continue to grow in parameter count, deploying them directly on mobile devices, edge hardware, or latency-sensitive production environments becomes increasingly difficult. Knowledge distillation is one of the leading approaches to resolving this tension — it lets us transfer the "wisdom" of a large model into a much lighter container.

Core Mechanisms of Knowledge Distillation

Soft Labels and the Temperature Parameter

Traditional model training uses hard labels — each sample has exactly one correct answer. Knowledge distillation's elegance lies in having the student model learn from the soft labels produced by the teacher model — that is, the probability distribution the teacher assigns across all classes.

For example, given an image of a cat, a teacher model might output a distribution like "cat 90%, dog 8%, fox 2%." This distribution carries far more information than the hard label "this is a cat": it tells the student model that cats and dogs are more similar in feature space, while other classes are more distant. This encoding of inter-class relationships is called dark knowledge, and it's the key reason knowledge distillation works.

To make soft labels smoother and expose more inter-class relationships, training typically introduces a temperature parameter to adjust the softmax output. A higher temperature produces a flatter distribution, allowing the student model to capture more fine-grained information.

Mathematically, the temperature parameter T is applied by dividing the input logits by T before computing the softmax: $p_i = \exp(z_i/T) / \sum_j \exp(z_j/T)$. When T=1, the output is the standard softmax distribution. When T>1, the distribution becomes more uniform — classes that were previously suppressed by low probabilities get amplified, enabling the student model to perceive relationships the teacher considers "unlikely but somewhat similar." After distillation training, the student model reverts to T=1 for standard inference. In practice, T is commonly set between 3 and 10, with the exact value tuned based on the task and model architecture.

Combining Loss Functions

The student model's training objective typically consists of two components: a standard loss against the true hard labels, and a distillation loss against the teacher's soft labels (usually measured by KL divergence, which quantifies the difference between two probability distributions). By combining these two loss terms with a weighting factor, the student model learns both the correct answers and the teacher's reasoning logic.

KL divergence (Kullback-Leibler Divergence) measures the "information gap" between two probability distributions: if the student's output distribution perfectly matches the teacher's soft labels, KL divergence is zero; the larger the difference, the higher the value. In the distillation loss, the teacher distribution serves as the reference, and deviations in the student distribution are penalized. The full training objective takes the form: $L = \alpha \cdot L_{CE}(y, p_s) + (1-\alpha) \cdot T^2 \cdot L_{KL}(p_t^T, p_s^T)$, where $\alpha$ is the weighting coefficient between the two loss terms, and $T^2$ is a gradient correction factor introduced by temperature scaling. The choice of $\alpha$ directly influences whether the student model prioritizes "memorizing correct answers" or "mimicking the teacher's reasoning" — making it one of the most important hyperparameters in distillation experiments.

Why Knowledge Distillation Works

The reason knowledge distillation allows small models to far outperform their "trained from scratch" counterparts comes down to the richness of the supervisory signal. Hard labels only provide "right or wrong" information, while soft labels convey the inter-class relationship map that the teacher model has distilled from massive training data.

This is analogous to a student who doesn't just know the correct answer, but also understands the reasoning behind it and the relative plausibility of other options. For a small model with limited parameters, this additional guidance greatly compensates for its expressive limitations — enabling it to maintain accuracy close to the large model while running several times (or even dozens of times) faster.

Real-World Applications

Knowledge distillation has seen widespread adoption in industry. Models deployed on mobile and embedded devices are typically distilled to fit within constrained compute and memory budgets. In natural language processing, distilled models like DistilBERT can retain most of the original BERT's performance with roughly half the parameters — a classic example of lightweight deployment.

In the era of large language models, knowledge distillation has become even more significant. Distilling the capabilities of a hundred-billion-parameter model down to a few billion parameters can dramatically reduce inference costs, making high-quality AI services accessible to more teams and individuals.

DistilBERT was introduced by Hugging Face in 2019, applying distillation directly during pre-training: using the original BERT-base (110M parameters) as the teacher, it trained a student model with only 66M parameters. Experiments showed that DistilBERT retained approximately 97% of the original model's performance on the GLUE benchmark, with roughly 60% faster inference and a 40% reduction in model size. This case demonstrated that distillation is not limited to supervised fine-tuning — it can also be applied during unsupervised pre-training, greatly expanding the technique's applicability. Subsequently, models like TinyBERT and MobileBERT have combined distillation with architecture search to push compression ratios even further.

Recommendations for Learning Knowledge Distillation

For those who want to develop a deep understanding of this technique, a three-layer approach is recommended: start by understanding how softmax and the temperature parameter shape soft labels; then implement a simple distillation loss function yourself; and finally compare the results of "distillation training" versus "independent training" on a real dataset. This progressive, hands-on approach tends to build stronger intuition than reading theory alone.

The original video creator also welcomed community feedback. Explanatory content made by practitioners tends to break down abstract concepts in more grounded, accessible ways — making it an excellent starting point for anyone new to this topic.

Conclusion

Knowledge distillation resolves the long-standing tension between performance and efficiency with an elegant idea: rather than training a small model from scratch, let it learn by standing on the shoulders of a large model. Understanding soft labels, the temperature parameter, and the combined loss function is all you need to grasp the essence of this technique. As model sizes continue to grow and deployment requirements become increasingly diverse, knowledge distillation will remain an indispensable tool in the model compression toolkit.