Q8_0
An 8-bit quantization format in which every 32 weights share a scaling factor, approaching lossless quality.
Core Facts
Timeline (last 90 days)
In tests on Qwen3.8 27B, the absolute KLD increase of the q4_0 KV cache harms more precise weights more: Q3-level weights increase by only 6-10%, UD-Q4_K_XL increases by about 39%, and Q8_0 increases by about 390%.
LTX-2.5 GGUF Q8_0 video is about 10% faster on NVIDIA GPUs, with completely identical frames.
Different quantization types (such as Q4_K_M and Q8_0) each have trade-offs in size, speed and quality.
Q8_0 is 8-bit block quantization with high precision but limited compression; Q4_K_M balances size and quality through mixed precision at the 4-bit level and is one of the commonly used default tiers for local deployment.
Running the same Q8_0 file twice, the second time with q8_0 K/V cache, gives KL 0.0038, a larger effect than quantizing weights from BF16 to Q8_0 (KL 0.0022).
The KV cache size can be expressed with the formula cache = 2 (K and V) × layers × KV heads × head dim × tokens × 2 bytes, where 2 bytes corresponds to FP16, doubling when using FP32, and halving when using INT8 quantization.
Q8_0 means 8-bit integer quantization: weights are stored in 8-bit blocks, each block carrying a scale factor, roughly halving compared with FP16, with very little precision loss but almost no memory savings.
Q8_0 quantization compresses model weights from 32-bit floating-point numbers to 8-bit integers, with precision loss typically within 1% compared to FP32.
In WikiText-2 GGUF runtime testing, Q8_0 quantization retained 99.1% of the BF16 reader NLL gain.
Choosing Q8_0 for context quantization means retaining higher precision on the KV Cache, which often has a more significant impact on inference quality than weight quantization.
3 more timeline events
All Facts (13)
The KV cache size can be expressed with the formula cache = 2 (K and V) × layers × KV heads × head dim × tokens × 2 bytes, where 2 bytes corresponds to FP16, doubling when using FP32, and halving when using INT8 quantization.
65%UnverifiedThe developer's quantization configuration for Qwen-3.8 uses Q4_K_S for weights and Q8_0 for context quantization.
60%UnverifiedIn tests on Qwen3.8 27B, the absolute KLD increase of the q4_0 KV cache harms more precise weights more: Q3-level weights increase by only 6-10%, UD-Q4_K_XL increases by about 39%, and Q8_0 increases by about 390%.
50%UnverifiedLTX-2.5 GGUF Q8_0 video is about 10% faster on NVIDIA GPUs, with completely identical frames.
50%UnverifiedDifferent quantization types (such as Q4_K_M and Q8_0) each have trade-offs in size, speed and quality.
50%UnverifiedQ8_0 is 8-bit block quantization with high precision but limited compression; Q4_K_M balances size and quality through mixed precision at the 4-bit level and is one of the commonly used default tiers for local deployment.
50%UnverifiedRunning the same Q8_0 file twice, the second time with q8_0 K/V cache, gives KL 0.0038, a larger effect than quantizing weights from BF16 to Q8_0 (KL 0.0022).
50%UnverifiedQ8_0 means 8-bit integer quantization: weights are stored in 8-bit blocks, each block carrying a scale factor, roughly halving compared with FP16, with very little precision loss but almost no memory savings.
50%UnverifiedQ8_0 quantization compresses model weights from 32-bit floating-point numbers to 8-bit integers, with precision loss typically within 1% compared to FP32.
50%UnverifiedIn WikiText-2 GGUF runtime testing, Q8_0 quantization retained 99.1% of the BF16 reader NLL gain.
50%UnverifiedChoosing Q8_0 for context quantization means retaining higher precision on the KV Cache, which often has a more significant impact on inference quality than weight quantization.
50%UnverifiedQ8_0 is an 8-bit integer quantization format, one of the highest-precision integer quantizations, and causes almost no perceptible degradation in output quality.
50%UnverifiedA 27B-parameter Q8_0 model, together with vLLM's KV cache and runtime overhead, requires nearly 50 GB of memory space for a single inference service.
50%