Model Quantization
Model quantization is a technique that converts neural network weights from high-precision floating-point representations to lower-precision formats, significantly reducing model memory footprint and accelerating inference.
Core Facts
Quantization technology compresses 32-bit or 16-bit floating-point weights into 4-bit or 8-bit integer representations, trading a small amount of precision for improvements in memory usage and speed.
A 70B-parameter FP16 model requires approximately 140 GB of video memory, approximately 70 GB after INT8 quantization, and approximately 35–40 GB after INT4 quantization.
Quantization technology compresses model weights from FP32/FP16 to low-precision formats such as INT8/INT4, which can reduce model size by 50%–75%.
Timeline (last 90 days)
Over the past two years, techniques such as model quantization, pruning, and knowledge distillation have allowed hundred-billion-parameter models to run on a single consumer-grade graphics card or even on a phone.
A 125-billion-parameter model can run on 12G of VRAM, relying mainly on model quantization and a mixture-of-experts (MoE) architecture, achieved by activating only part of the expert parameters, quantizing weights to 4bit or lower, and dynamically loading inactive parameters from memory or SSD on demand.
The model backbone is typically quantized to INT8, INT4 or other low-bit formats to save memory and speed up inference.
The estimation formula for model memory usage is approximately: memory usage (GB) ≈ bit-width / 8 × number of parameters (in billions).
Traditional model quantization takes the path of reducing the numerical precision of weights, evolving from FP16 to INT8 and then to 4-bit, 2-bit, and even 1-bit.
Choosing a quantization level is essentially a trade-off between VRAM budget and output quality, with no universally optimal solution.
A larger base model, even after aggressive quantization, still outperforms a smaller-parameter model with light quantization in overall capability.
The technical foundation of local LLM execution is model quantization.
Vendors can apply quantization compression to models (reducing weights from 16-bit floating point to 8-bit or even 4-bit integers to save VRAM), substitute distilled models for some requests, or adjust safety filter thresholds without changing the API endpoint name, all of which are invisible to the caller.
As computing costs decline and inference acceleration technologies such as model quantization and speculative decoding become widespread, the generation speed for equivalent quality is expected to be significantly reduced within the next one to two years.
16 more timeline events
All Facts (20)
Quantization technology compresses 32-bit or 16-bit floating-point weights into 4-bit or 8-bit integer representations, trading a small amount of precision for improvements in memory usage and speed.
90%VerifiedA 70B-parameter FP16 model requires approximately 140 GB of video memory, approximately 70 GB after INT8 quantization, and approximately 35–40 GB after INT4 quantization.
90%VerifiedQuantization technology compresses model weights from FP32/FP16 to low-precision formats such as INT8/INT4, which can reduce model size by 50%–75%.
90%VerifiedThe essence of model quantization is to map a model's floating-point weights and activations (typically FP32) to low-bit-width representations (such as INT8 or INT4), thereby reducing memory usage, improving inference speed, and lowering power consumption.
85%Verified4-bit quantization reduces model size to 1/8 of the original at the cost of approximately 1–3% accuracy loss.
80%Verified模型蒸馏是将大模型的能力压缩到小模型中以降低推理成本的技术
80%VerifiedModel quantization techniques (such as 4-bit/8-bit quantization in GGUF format) enable large models that originally required tens of GB of VRAM to run on consumer-grade GPUs or even CPUs.
80%VerifiedWhen speed improvements and price reductions occur simultaneously, they typically point to underlying inference architecture optimization, with possible paths including model distillation, quantization, speculative decoding, and streaming processing architectures.
75%VerifiedModel quantization replaces high-precision parameters with lower-precision numerical representations, and can theoretically reduce model size and computation by 2–4×.
75%VerifiedFor the 4-bit quantized version of a 7B parameter model, at least 8GB of memory and a modern multi-core CPU are typically required for smooth operation, with inference speeds of approximately 10–30 tokens per second
75%VerifiedFP8 quantization can compress the weights of a 7B model from approximately 14GB (FP16) to approximately 7GB.
70%Verified量化技术可将模型体积压缩50%-75%，在损失极小精度前提下降低显存占用
65%Verified模型量化是将神经网络权重从高精度浮点数压缩为低精度整数的技术
65%Unverified推测解码、模型蒸馏、KV Cache优化等推理优化技术的持续进步正在使单位推理成本快速下降
85%Partially VerifiedA higher acceptance rate means the draft model's guesses are more accurate and the large model needs to redo less work, which directly determines the actual speedup achieved by speculative decoding.
80%Unverified量化模型的困惑度可能仅上升0.1-0.5，但在特定下游任务（如代码生成、数学推理）上的表现可能出现显著退化
75%UnverifiedThe main optimization approaches in the industry for reducing LLM inference costs include model quantization, speculative decoding, and semantic caching mechanisms.
60%UnverifiedQuantization-Aware Training (QAT) lets the model retain a full-precision copy of weights, practicing rounding operations in each forward pass to learn weight distributions under extreme compression.
60%UnverifiedThe output quality of 4-bit quantized models can typically reach more than 95% of that of the original model.
60%UnverifiedOver the past two years, techniques such as model quantization, pruning, and knowledge distillation have allowed hundred-billion-parameter models to run on a single consumer-grade graphics card or even on a phone.
50%