GPTQ
A post-training quantization method proposed in 2022 that uses layer-wise second-order optimization to compress large models to INT4 precision, representing a significant milestone in large language model quantization.
Core Facts
GPTQ uses second-order gradient information to minimize quantization error, AWQ identifies and protects the weights with the greatest influence on activation values, and GGUF was introduced by the llama.cpp project and is specifically optimized for CPU inference.
Common model quantization methods include formats such as GPTQ, AWQ, and GGUF.
AWQ (2023) found that approximately 1% of salient channels in the weights have the greatest impact on model output, and protects these channels through equivalent scaling.
Timeline (last 90 days)
A 125-billion-parameter model can run on 12G of VRAM, relying mainly on model quantization and a mixture-of-experts (MoE) architecture, achieved by activating only part of the expert parameters, quantizing weights to 4bit or lower, and dynamically loading inactive parameters from memory or SSD on demand.
vLLM natively supports AWQ and GPTQ quantization, which can be specified via --quantization awq.
Choosing a local inference backend requires comprehensive consideration of the supported model formats (GGUF, GPTQ, etc.) and their compatibility with the target hardware.
AWQ, GPTQ, and FP8 are formats specifically optimized for CUDA GPUs and cannot run efficiently in a CPU-only environment
For running high-concurrency vLLM on a server, choose AWQ, GPTQ, or FP8 quantized versions, or the official BF16 original
GPTQ uses second-order gradient information to reconstruct weights layer by layer.
The input matrices that the vLLM linear backend must handle may be in FP16, BF16, INT8, or various GPTQ/AWQ quantization formats.
The three 4-bit quantization schemes—RTN, GPTQ, and AWQ—have similar compression ratios but progressively increasing precision retention capabilities.
The study tested three 4-bit quantization schemes—RTN, GPTQ, and AWQ—and no configuration fell out of or entered the qualification range due to quantization.
NVFP4 is typically used in conjunction with fine-grained scaling factors (per-group scaling) and activation-aware quantization (AWQ/GPTQ) to offset quantization error.
40 more timeline events
All Facts (20)
GPTQ uses second-order gradient information to minimize quantization error, AWQ identifies and protects the weights with the greatest influence on activation values, and GGUF was introduced by the llama.cpp project and is specifically optimized for CPU inference.
90%VerifiedCommon model quantization methods include formats such as GPTQ, AWQ, and GGUF.
90%VerifiedAWQ (2023) found that approximately 1% of salient channels in the weights have the greatest impact on model output, and protects these channels through equivalent scaling.
85%Verified4-bit quantization reduces model size to 1/8 of the original at the cost of approximately 1–3% accuracy loss.
80%VerifiedGGUF is a model format standard introduced by the llama.cpp project, supporting multiple quantization levels ranging from Q2_K to Q8_0.
80%Verified4-bit quantization compresses each parameter to 0.5 bytes, theoretically compressing 140GB down to approximately 35–40GB.
80%VerifiedAWQ identifies and protects the weights with the greatest influence on activation values, and typically achieves better accuracy than GPTQ at the same compression ratio.
80%VerifiedThe core insight of AWQ is that only approximately 1% of salient weights have a major impact on model output, and protecting them enables model quality to be maintained at extremely low bit widths
75%VerifiedINT4 quantization can theoretically compress a model's VRAM footprint to one-quarter of FP16, with commonly used tools including GPTQ, AWQ, and GGUF.
75%VerifiedCommon quantization schemes include GPTQ (requires GPU), AWQ, and GGUF format quantization within the llama.cpp ecosystem.
65%VerifiedModern quantization schemes such as GPTQ and AWQ introduce layer-by-layer calibration and error compensation mechanisms to keep perplexity loss within 1–2%
65%VerifiedStacking techniques such as K-quant, GPTQ, and AWQ can smoothly extend quantization to 4-bit, but performance catastrophically collapses when pushed to 2-bit or 1-bit.
65%VerifiedOllama uses the GGUF format by default; other common quantization formats include GPTQ.
65%VerifiedModern quantization schemes such as GPTQ, AWQ, and the grouped quantization of the GGUF format can reduce model size by more than 75% while retaining the vast majority of reasoning capability through layer-by-layer calibration and outlier protection.
65%VerifiedGPTQ is a post-training quantization scheme based on layer-wise error compensation, with minimal precision loss.
65%VerifiedMainstream quantization schemes include GPTQ, AWQ, and GGML's Q4_K_M, each making different trade-offs between compression ratio and precision loss.
65%Verified量化技术可将模型体积压缩50%-75%，在损失极小精度前提下降低显存占用
65%Verified社区主流量化格式包括GGUF、AWQ和GPTQ，其中GGUF由llama.cpp项目推广并支持CPU推理
65%VerifiedModel quantization technologies (GGUF, GPTQ, AWQ, and other formats) enable a 70B parameter model to run on a consumer-grade computer with 32GB of memory
65%Verified4-bit量化技术（如GPTQ、AWQ、GGUF格式）可以将模型显存需求降低到原来的1/4，但会带来一定精度损失
65%