Q4 Quantization
A technique that compresses model weights into approximately 4-bit integer representations. It can significantly reduce memory usage and accelerate inference, and is often combined with mixed-precision strategies.
Core Facts
Q4 quantization compresses each weight parameter from 16 bits to 4 bits, theoretically reducing model size by approximately 75%
Q4 means each weight is stored using 4-bit integers, reducing the size by approximately 75% compared to the original 16-bit floating point
Timeline (last 90 days)
The weights of a Q4-quantized model are roughly 30% the size of BF16 precision, a ratio that accounts for quantization scaling factors and redundant storage overhead.
On a desktop CPU of similar performance, the generation speed of a 7B Q4 quantized model is roughly between a few tokens and a dozen-plus tokens per second.
Q4 means each weight is stored using 4-bit integers, reducing the size by approximately 75% compared to the original 16-bit floating point
Q4 quantized weights are approximately 111GB, which neither VRAM nor RAM can fully accommodate, requiring missing experts to be transferred from the SSD through RAM into VRAM, with the bottleneck being bandwidth rather than compute.
Using Q4 quantized weights (a modified version of llama.cpp), the generation speed on the first run was only 1.83 tokens per second.
The three-level cache improved generation speed under Q4 weights from 1.83 tokens per second to 8.89, an improvement of nearly 5x.
A 70B-parameter model requires approximately 35–40GB of VRAM after Q4 quantization, which a single T4 cannot handle, but dual T4s can basically satisfy.
Ollama downloads Q4 quantized models by default, which have small VRAM usage but lower precision and more pronounced hallucination behavior.
Q4 quantization is sufficient for casual conversation and code completion, but for scenarios involving rigorous analysis, long document processing, or the need to reduce hallucinations, switching to Q6 or Q8 versions is recommended.
Compared to Q4 quantization, Q8 quantization retains higher precision closer to the original model quality, but consumes more VRAM.
All Facts (11)
Q4 quantization compresses each weight parameter from 16 bits to 4 bits, theoretically reducing model size by approximately 75%
90%VerifiedQ4 means each weight is stored using 4-bit integers, reducing the size by approximately 75% compared to the original 16-bit floating point
70%UnverifiedCompared to Q4 quantization, Q8 quantization retains higher precision closer to the original model quality, but consumes more VRAM.
60%UnverifiedThe weights of a Q4-quantized model are roughly 30% the size of BF16 precision, a ratio that accounts for quantization scaling factors and redundant storage overhead.
50%UnverifiedOn a desktop CPU of similar performance, the generation speed of a 7B Q4 quantized model is roughly between a few tokens and a dozen-plus tokens per second.
50%UnverifiedUsing Q4 quantized weights (a modified version of llama.cpp), the generation speed on the first run was only 1.83 tokens per second.
50%UnverifiedQ4 quantized weights are approximately 111GB, which neither VRAM nor RAM can fully accommodate, requiring missing experts to be transferred from the SSD through RAM into VRAM, with the bottleneck being bandwidth rather than compute.
50%UnverifiedThe three-level cache improved generation speed under Q4 weights from 1.83 tokens per second to 8.89, an improvement of nearly 5x.
50%UnverifiedA 70B-parameter model requires approximately 35–40GB of VRAM after Q4 quantization, which a single T4 cannot handle, but dual T4s can basically satisfy.
50%UnverifiedQ4 quantization is sufficient for casual conversation and code completion, but for scenarios involving rigorous analysis, long document processing, or the need to reduce hallucinations, switching to Q6 or Q8 versions is recommended.
50%UnverifiedOllama downloads Q4 quantized models by default, which have small VRAM usage but lower precision and more pronounced hallucination behavior.
50%