Q4 Quantization
A technique that compresses model weights into approximately 4-bit integer representations. It can significantly reduce memory usage and accelerate inference, and is often combined with mixed-precision strategies.
Core Facts
Timeline (last 90 days)
The weights of a Q4-quantized model are roughly 30% the size of BF16 precision, a ratio that accounts for quantization scaling factors and redundant storage overhead.
On a desktop CPU of similar performance, the generation speed of a 7B Q4 quantized model is roughly between a few tokens and a dozen-plus tokens per second.
Using Q4 quantized weights (a modified version of llama.cpp), the generation speed on the first run was only 1.83 tokens per second.
Q4 quantized weights are approximately 111GB, which neither VRAM nor RAM can fully accommodate, requiring missing experts to be transferred from the SSD through RAM into VRAM, with the bottleneck being bandwidth rather than compute.
The three-level cache improved generation speed under Q4 weights from 1.83 tokens per second to 8.89, an improvement of nearly 5x.
A 70B-parameter model requires approximately 35–40GB of VRAM after Q4 quantization, which a single T4 cannot handle, but dual T4s can basically satisfy.
Q4 quantization is sufficient for casual conversation and code completion, but for scenarios involving rigorous analysis, long document processing, or the need to reduce hallucinations, switching to Q6 or Q8 versions is recommended.
Ollama downloads Q4 quantized models by default, which have small VRAM usage but lower precision and more pronounced hallucination behavior.
Compared to Q4 quantization, Q8 quantization retains higher precision closer to the original model quality, but consumes more VRAM.
All Facts (10)
Q4量化将每个权重参数从16位压缩到4位，理论上可将模型体积缩小约75%
65%UnverifiedThe weights of a Q4-quantized model are roughly 30% the size of BF16 precision, a ratio that accounts for quantization scaling factors and redundant storage overhead.
50%UnverifiedOn a desktop CPU of similar performance, the generation speed of a 7B Q4 quantized model is roughly between a few tokens and a dozen-plus tokens per second.
50%UnverifiedThe three-level cache improved generation speed under Q4 weights from 1.83 tokens per second to 8.89, an improvement of nearly 5x.
50%UnverifiedUsing Q4 quantized weights (a modified version of llama.cpp), the generation speed on the first run was only 1.83 tokens per second.
50%UnverifiedQ4 quantized weights are approximately 111GB, which neither VRAM nor RAM can fully accommodate, requiring missing experts to be transferred from the SSD through RAM into VRAM, with the bottleneck being bandwidth rather than compute.
50%UnverifiedA 70B-parameter model requires approximately 35–40GB of VRAM after Q4 quantization, which a single T4 cannot handle, but dual T4s can basically satisfy.
50%UnverifiedOllama downloads Q4 quantized models by default, which have small VRAM usage but lower precision and more pronounced hallucination behavior.
50%UnverifiedQ4 quantization is sufficient for casual conversation and code completion, but for scenarios involving rigorous analysis, long document processing, or the need to reduce hallucinations, switching to Q6 or Q8 versions is recommended.
50%UnverifiedCompared to Q4 quantization, Q8 quantization retains higher precision closer to the original model quality, but consumes more VRAM.
50%