Distillation

Model Distillation in machine learning refers to training a small student model using the outputs of a large teacher model, in order to compress model size while retaining capability.

The teacher model in distillation does not have to be larger than the student model; it can be a domain expert model of comparable size.

Flash lightweight models reduce inference costs and improve response speed through smaller parameter sizes, quantization, or distillation techniques.

The distilled version of DeepSeek-R1 is a successful example of distillation technology; quantized or distilled versions can be deployed on consumer-grade GPUs or even laptops.

Migrating a model from cloud inference to local operation typically requires compression steps such as quantization, pruning and distillation.

Against the backdrop of compute scarcity, the priority of model compression techniques such as quantization and distillation has been significantly moved forward.

Although SlimWise's low-cost distillation stage is helpful, it adds an extra training step before deployment, requiring additional compute and data.

The paper points out that the ceiling of distillation effectiveness is determined not only by the teacher's parameter scale, but more by the quality of the supervision signal at the edges of the distribution.

Anthropic disclosed to investors that the distillation business contributed 5% to 7% of total revenue.

Open-source weights mean the model parameter files can be downloaded and run on your own; distillation is using a large model to generate data to train a model with far fewer parameters, and distilled small models commonly come in sizes of 7B, 14B and 32B, and after quantization and compression can run on a single consumer-grade graphics card or even purely on CPU.

Streaming loading schemes can be used in combination with techniques such as quantization, distillation, and KV Cache optimization; quantization shrinks the size of a single expert, while streaming loading solves the total-volume problem.

Quantization and distillation combined often reduce inference cost per token by several times.

The large model algorithm research direction focuses on foundational research within enterprises, including writing operators, optimizing the large model itself, and optimizing training and inference processes.

In the diffusion model ecosystem, Turbo typically means reducing the number of sampling steps through techniques such as distillation, allowing images to form in fewer iterations.

When distillation techniques compress the number of steps to an extreme degree, the richness of image detail and the fidelity of certain complex scenes may decline slightly.

Distillation, consistency training, and adversarial objectives are the mainstream approaches to few-step generation, but they sacrifice the likelihood framework in the process of compressing the number of steps

Distillation, quantization, and inference scheduling improvements can significantly reduce per-token serving costs, which is a relatively independent curve from intelligence improvement.

Everything that can be learned through RL is essentially a small number of bits that can be easily distilled and replicated, which is the primary force countering the centralization of model providers.

Distillation can be decomposed into four dimensions: hard labels/soft labels, data source, offline/online timing, and teacher identity.

The phenomenon of smaller models outperforming larger ones suggests that the two companies may have trained independently at different scales, rather than taking the route of distilling large models into small ones.