AWQ
A weight quantization technique that analyzes activation value distributions to protect critical weights, achieving lower precision loss compared to other quantization approaches at equivalent bit widths.
Core Facts
GPTQ uses second-order gradient information to minimize quantization error, AWQ identifies and protects the weights with the greatest influence on activation values, and GGUF was introduced by the llama.cpp project and is specifically optimized for CPU inference.
AWQ (Activation-aware Weight Quantization) technology identifies critical weight channels for special protection by observing activation values during model inference, and compresses the remaining weights to 4-bit precision.
Common model quantization methods include formats such as GPTQ, AWQ, and GGUF.
Timeline (last 90 days)
A 125-billion-parameter model can run on 12G of VRAM, relying mainly on model quantization and a mixture-of-experts (MoE) architecture, achieved by activating only part of the expert parameters, quantizing weights to 4bit or lower, and dynamically loading inactive parameters from memory or SSD on demand.
vLLM natively supports AWQ and GPTQ quantization, which can be specified via --quantization awq.
Methods such as SmoothQuant and AWQ mitigate quantization accuracy loss by applying scaling migration to outlier channels or protecting them with mixed precision.
AWQ, GPTQ, and FP8 are formats specifically optimized for CUDA GPUs and cannot run efficiently in a CPU-only environment
For running high-concurrency vLLM on a server, choose AWQ, GPTQ, or FP8 quantized versions, or the official BF16 original
P106无法使用FlashAttention、AWQ等主流加速方案
AWQ (Activation-aware Weight Quantization) protects critical weights by analyzing activation value distributions.
The input matrices that the vLLM linear backend must handle may be in FP16, BF16, INT8, or various GPTQ/AWQ quantization formats.
The study tested three 4-bit quantization schemes—RTN, GPTQ, and AWQ—and no configuration fell out of or entered the qualification range due to quantization.
The three 4-bit quantization schemes—RTN, GPTQ, and AWQ—have similar compression ratios but progressively increasing precision retention capabilities.
40 more timeline events
All Facts (20)
GPTQ uses second-order gradient information to minimize quantization error, AWQ identifies and protects the weights with the greatest influence on activation values, and GGUF was introduced by the llama.cpp project and is specifically optimized for CPU inference.
90%VerifiedAWQ (Activation-aware Weight Quantization) technology identifies critical weight channels for special protection by observing activation values during model inference, and compresses the remaining weights to 4-bit precision.
90%VerifiedCommon model quantization methods include formats such as GPTQ, AWQ, and GGUF.
90%VerifiedAWQ (2023) found that approximately 1% of salient channels in the weights have the greatest impact on model output, and protects these channels through equivalent scaling.
85%Verified4-bit quantization reduces model size to 1/8 of the original at the cost of approximately 1–3% accuracy loss.
80%VerifiedGGUF is a model format standard introduced by the llama.cpp project, supporting multiple quantization levels ranging from Q2_K to Q8_0.
80%Verified4-bit quantization compresses each parameter to 0.5 bytes, theoretically compressing 140GB down to approximately 35–40GB.
80%VerifiedAWQ identifies and protects the weights with the greatest influence on activation values, and typically achieves better accuracy than GPTQ at the same compression ratio.
80%VerifiedExtreme 4-bit quantization schemes such as GPTQ and AWQ enable 70B-parameter large models to run on consumer-grade GPUs.
80%VerifiedThe core insight of AWQ is that only approximately 1% of salient weights have a major impact on model output, and protecting them enables model quality to be maintained at extremely low bit widths
75%VerifiedINT4 quantization can theoretically compress a model's VRAM footprint to one-quarter of FP16, with commonly used tools including GPTQ, AWQ, and GGUF.
75%VerifiedCommon quantization schemes include GPTQ (requires GPU), AWQ, and GGUF format quantization within the llama.cpp ecosystem.
65%VerifiedModern quantization schemes such as GPTQ and AWQ introduce layer-by-layer calibration and error compensation mechanisms to keep perplexity loss within 1–2%
65%VerifiedStacking techniques such as K-quant, GPTQ, and AWQ can smoothly extend quantization to 4-bit, but performance catastrophically collapses when pushed to 2-bit or 1-bit.
65%VerifiedOllama uses the GGUF format by default; other common quantization formats include GPTQ.
65%VerifiedModern quantization schemes such as GPTQ, AWQ, and the grouped quantization of the GGUF format can reduce model size by more than 75% while retaining the vast majority of reasoning capability through layer-by-layer calibration and outlier protection.
65%VerifiedMainstream quantization schemes include GPTQ, AWQ, and GGML's Q4_K_M, each making different trade-offs between compression ratio and precision loss.
65%Verified量化技术可将模型体积压缩50%-75%，在损失极小精度前提下降低显存占用
65%Verified社区主流量化格式包括GGUF、AWQ和GPTQ，其中GGUF由llama.cpp项目推广并支持CPU推理
65%VerifiedModel quantization technologies (GGUF, GPTQ, AWQ, and other formats) enable a 70B parameter model to run on a consumer-grade computer with 32GB of memory
65%