[KongchangAI]
Concept

Q8_0

An 8-bit quantization format in which every 32 weights share a scaling factor, approaching lossless quality.

Core Facts

Timeline (last 90 days)

Nov 7

In tests on Qwen3.8 27B, the absolute KLD increase of the q4_0 KV cache harms more precise weights more: Q3-level weights increase by only 6-10%, UD-Q4_K_XL increases by about 39%, and Q8_0 increases by about 390%.

Unverified50%
Nov 6

LTX-2.5 GGUF Q8_0 video is about 10% faster on NVIDIA GPUs, with completely identical frames.

Unverified50%
Oct 28

Different quantization types (such as Q4_K_M and Q8_0) each have trade-offs in size, speed and quality.

Unverified50%
Oct 22

Q8_0 is 8-bit block quantization with high precision but limited compression; Q4_K_M balances size and quality through mixed precision at the 4-bit level and is one of the commonly used default tiers for local deployment.

Unverified50%
Oct 17

Running the same Q8_0 file twice, the second time with q8_0 K/V cache, gives KL 0.0038, a larger effect than quantizing weights from BF16 to Q8_0 (KL 0.0022).

Unverified50%
Oct 16

The KV cache size can be expressed with the formula cache = 2 (K and V) × layers × KV heads × head dim × tokens × 2 bytes, where 2 bytes corresponds to FP16, doubling when using FP32, and halving when using INT8 quantization.

Verified65%
Oct 15

Q8_0 means 8-bit integer quantization: weights are stored in 8-bit blocks, each block carrying a scale factor, roughly halving compared with FP16, with very little precision loss but almost no memory savings.

Unverified50%
Oct 10

Q8_0 quantization compresses model weights from 32-bit floating-point numbers to 8-bit integers, with precision loss typically within 1% compared to FP32.

Unverified50%
Sep 25

In WikiText-2 GGUF runtime testing, Q8_0 quantization retained 99.1% of the BF16 reader NLL gain.

Unverified50%
Sep 24

Choosing Q8_0 for context quantization means retaining higher precision on the KV Cache, which often has a more significant impact on inference quality than weight quantization.

Unverified50%

3 more timeline events

All Facts (13)

Verified

The KV cache size can be expressed with the formula cache = 2 (K and V) × layers × KV heads × head dim × tokens × 2 bytes, where 2 bytes corresponds to FP16, doubling when using FP32, and halving when using INT8 quantization.

65%
Unverified

The developer's quantization configuration for Qwen-3.8 uses Q4_K_S for weights and Q8_0 for context quantization.

60%
Unverified

In tests on Qwen3.8 27B, the absolute KLD increase of the q4_0 KV cache harms more precise weights more: Q3-level weights increase by only 6-10%, UD-Q4_K_XL increases by about 39%, and Q8_0 increases by about 390%.

50%
Unverified

LTX-2.5 GGUF Q8_0 video is about 10% faster on NVIDIA GPUs, with completely identical frames.

50%
Unverified

Different quantization types (such as Q4_K_M and Q8_0) each have trade-offs in size, speed and quality.

50%
Unverified

Q8_0 is 8-bit block quantization with high precision but limited compression; Q4_K_M balances size and quality through mixed precision at the 4-bit level and is one of the commonly used default tiers for local deployment.

50%
Unverified

Running the same Q8_0 file twice, the second time with q8_0 K/V cache, gives KL 0.0038, a larger effect than quantizing weights from BF16 to Q8_0 (KL 0.0022).

50%
Unverified

Q8_0 means 8-bit integer quantization: weights are stored in 8-bit blocks, each block carrying a scale factor, roughly halving compared with FP16, with very little precision loss but almost no memory savings.

50%
Unverified

Q8_0 quantization compresses model weights from 32-bit floating-point numbers to 8-bit integers, with precision loss typically within 1% compared to FP32.

50%
Unverified

In WikiText-2 GGUF runtime testing, Q8_0 quantization retained 99.1% of the BF16 reader NLL gain.

50%
Unverified

Choosing Q8_0 for context quantization means retaining higher precision on the KV Cache, which often has a more significant impact on inference quality than weight quantization.

50%
Unverified

Q8_0 is an 8-bit integer quantization format, one of the highest-precision integer quantizations, and causes almost no perceptible degradation in output quality.

50%
Unverified

A 27B-parameter Q8_0 model, together with vLLM's KV cache and runtime overhead, requires nearly 50 GB of memory space for a single inference service.

50%

Source Articles