[KongchangAI]
ConceptQ4量化 / 4比特量化 / INT4量化

Q4 Quantization

A technique that compresses model weights into approximately 4-bit integer representations. It can significantly reduce memory usage and accelerate inference, and is often combined with mixed-precision strategies.

Core Facts

Timeline (last 90 days)

Oct 18

The weights of a Q4-quantized model are roughly 30% the size of BF16 precision, a ratio that accounts for quantization scaling factors and redundant storage overhead.

Unverified50%
Oct 16

On a desktop CPU of similar performance, the generation speed of a 7B Q4 quantized model is roughly between a few tokens and a dozen-plus tokens per second.

Unverified50%
Oct 7

Q4 means each weight is stored using 4-bit integers, reducing the size by approximately 75% compared to the original 16-bit floating point

Verified70%
Oct 5

Q4 quantized weights are approximately 111GB, which neither VRAM nor RAM can fully accommodate, requiring missing experts to be transferred from the SSD through RAM into VRAM, with the bottleneck being bandwidth rather than compute.

Unverified50%
Oct 5

Using Q4 quantized weights (a modified version of llama.cpp), the generation speed on the first run was only 1.83 tokens per second.

Unverified50%
Oct 5

The three-level cache improved generation speed under Q4 weights from 1.83 tokens per second to 8.89, an improvement of nearly 5x.

Unverified50%
Sep 16

A 70B-parameter model requires approximately 35–40GB of VRAM after Q4 quantization, which a single T4 cannot handle, but dual T4s can basically satisfy.

Unverified50%
Sep 16

Ollama downloads Q4 quantized models by default, which have small VRAM usage but lower precision and more pronounced hallucination behavior.

Unverified50%
Sep 16

Q4 quantization is sufficient for casual conversation and code completion, but for scenarios involving rigorous analysis, long document processing, or the need to reduce hallucinations, switching to Q6 or Q8 versions is recommended.

Unverified50%
Sep 12

Compared to Q4 quantization, Q8 quantization retains higher precision closer to the original model quality, but consumes more VRAM.

Unverified60%

All Facts (11)

Verified

Q4 quantization compresses each weight parameter from 16 bits to 4 bits, theoretically reducing model size by approximately 75%

90%
Verified

Q4 means each weight is stored using 4-bit integers, reducing the size by approximately 75% compared to the original 16-bit floating point

70%
Unverified

Compared to Q4 quantization, Q8 quantization retains higher precision closer to the original model quality, but consumes more VRAM.

60%
Unverified

The weights of a Q4-quantized model are roughly 30% the size of BF16 precision, a ratio that accounts for quantization scaling factors and redundant storage overhead.

50%
Unverified

On a desktop CPU of similar performance, the generation speed of a 7B Q4 quantized model is roughly between a few tokens and a dozen-plus tokens per second.

50%
Unverified

Using Q4 quantized weights (a modified version of llama.cpp), the generation speed on the first run was only 1.83 tokens per second.

50%
Unverified

Q4 quantized weights are approximately 111GB, which neither VRAM nor RAM can fully accommodate, requiring missing experts to be transferred from the SSD through RAM into VRAM, with the bottleneck being bandwidth rather than compute.

50%
Unverified

The three-level cache improved generation speed under Q4 weights from 1.83 tokens per second to 8.89, an improvement of nearly 5x.

50%
Unverified

A 70B-parameter model requires approximately 35–40GB of VRAM after Q4 quantization, which a single T4 cannot handle, but dual T4s can basically satisfy.

50%
Unverified

Q4 quantization is sufficient for casual conversation and code completion, but for scenarios involving rigorous analysis, long document processing, or the need to reduce hallucinations, switching to Q6 or Q8 versions is recommended.

50%
Unverified

Ollama downloads Q4 quantized models by default, which have small VRAM usage but lower precision and more pronounced hallucination behavior.

50%

Source Articles