INT4

INT4 quantization can theoretically compress a model's VRAM footprint to one-quarter of FP16, with commonly used tools including GPTQ, AWQ, and GGUF.

The core principle of quantization is to convert model parameters from high-precision floating-point representations (such as FP32's 32-bit floating point) to low-precision representations (such as INT8's 8-bit integer or INT4's 4-bit integer).

Re-quantizing INT4 weights to NF4 subjects each weight to two rounds of rounding, and official measurements show this increases the perplexity of Qwen3-8B by 1.6%

The A100 is also Ampere architecture, so running FP8-quantized models on the A100 is also inefficient, and an INT4 quantization scheme should be chosen.

Commonly used inference frameworks include vLLM, TensorRT-LLM, Ollama, etc., while quantization commonly uses methods such as INT8 and INT4 to compress model size.

A common industry practice is to quantize weights to INT4 or INT8 while retaining higher precision for critical layers, in order to strike a balance between cost and quality.

Unsloth's Triton kernels dequantize layer by layer on demand, keeping INT4 precise without re-quantizing to NF4, avoiding an approximately 1.6% increase in perplexity.

Quantization precision recommendations: prioritize BF16 for production environments, INT4 is acceptable for test environments, and the minimum threshold should not go below INT4, as lower precision results in greater loss of model capability and output quality.

Under FP4/INT4 quantization, the model size is approximately one-quarter of that under FP16.

The difference between NVFP4 and INT4 is that NVFP4 retains the dynamic range expressiveness of floating-point numbers, making it more friendly to large models with non-uniform weight distributions.

When using INT4 HQQ quantization, catastrophic transcription loops occurring when the model processes West African accents were amplified by 5 to 7 times.

In edge device and low-precision inference scenarios (such as INT8/INT4 quantization), exponential operations are often a concentrated source of quantization error and computational overhead

Using a quantization combination of INT8 weights with INT4 embedding layers in ONNX, the F1 score difference is only approximately 0.005 compared to more conservatively quantized versions.

INT4 quantization solutions including GPTQ and AWQ are suitable for scenarios that are extremely cost-sensitive and can tolerate a slight drop in quality.

Regardless of the encoding scheme used, 4-bit precision can only distinguish 16 different numerical levels.

When evaluating quantization schemes, one should look at the number of qualified outputs per unit time rather than simply single-inference speed, as the speed gains from INT4 may be offset by the overhead of repeated regeneration.

A model with 2B parameters typically requires only about 1–2 GB of memory after INT4 quantization, enabling it to run on mainstream smartphones, embedded development boards, and similar devices.