Pruning

Quantization, pruning, and low-rank decomposition are three common means of model compression.

Distillation, quantization, and pruning are often collectively called the three-piece model compression toolkit, but distillation changes the training signal, while the latter two directly compress existing weights.

Pruning compresses model size by removing neurons or attention heads that contribute little to the output.

Pruning streamlines the network structure by removing weights or neurons that contribute less to the output.

Over the past two years, techniques such as model quantization, pruning, and knowledge distillation have allowed hundred-billion-parameter models to run on a single consumer-grade graphics card or even on a phone.

剪枝和量化在保持原架构拓扑的前提下做减法，不改变层的功能语义和连接模式；而Priming允许目标架构在层数、宽度、注意力模式上与原模型不同。

Pruning is the process of identifying and removing neural network weights or entire channels that contribute minimally to the output, thereby reducing the model's structure.

All three approaches—pruning, quantization, and distillation—commonly face the problem of significant accuracy degradation after compression, requiring large-scale retraining to recover performance.

Pruning, quantization, and distillation are currently the three most mainstream model compression techniques.