KV Cache Quantization Benchmark: 413 Configurations Reveal KVarN 6-bit Outperforming q8_0

Why KV Cache Quantization Matters

When deploying large language models locally, the KV cache (Key-Value Cache) is often a hidden killer of VRAM consumption. The KV cache is a core component of the self-attention mechanism in the Transformer architecture—during autoregressive generation, every new token the model generates requires attention computation against all historical tokens. Without a KV cache, each step would need to recompute the Key and Value vectors for all historical tokens, resulting in O(n²) computational complexity. The KV cache avoids redundant computation by storing previously calculated Key-Value pairs, reducing the incremental computation per step to O(n). However, the trade-off is memory usage: for a model with L layers, H attention heads, and per-head dimension d, caching n tokens requires 2×L×H×d×n×bytes_per_precision in VRAM.

As context length increases, KV cache memory consumption grows linearly. Especially in 64K or even longer context scenarios, the KV cache can exceed the memory footprint of the model weights themselves. Take Qwen 27B as an example—at BF16 precision with 64K context, the KV cache can easily consume tens of gigabytes of VRAM. Therefore, how to compress the KV cache while preserving model output quality has become one of the core challenges in local inference optimization.

A recent large-scale benchmark from the Reddit community provides what is arguably the most systematic answer to date. The tester used BeeLlama.cpp v0.4.0 (a llama.cpp fork offering additional KV cache quantization options) to run 413 configurations across Qwen 3.6 27B and Gemma 4 31B—238 configurations for Qwen and 175 for Gemma.

The value of this test lies in its scale and methodological rigor—rather than relying on subjective "feels usable" assessments, it uses KL divergence (KLD) as an objective quantitative metric, directly measuring how much the quantized output distribution deviates from the BF16 reference distribution.

Core Metrics and New Methods Explained

Measuring True Quality Loss with KL Divergence

The test uses KL divergence (Kullback-Leibler Divergence) as its core evaluation metric. KL divergence originates from information theory—for discrete distributions P and Q, it is defined as KLD(P||Q) = Σ P(x) × log(P(x)/Q(x)). In this test scenario, P is the output logits distribution of the full-precision BF16 model, and Q is the output distribution of the quantized model. KLD = 0 means the two distributions are identical; higher values indicate greater deviation.

Compared to traditional metrics like perplexity, KLD's advantage is that it directly compares probability distribution deformation at the token level, capturing subtle biases introduced by quantization with greater sensitivity. KLD measures how much the quantized model's output probability distribution deviates from the full-precision BF16 baseline: lower KLD means less information loss from quantization and output quality closer to the original model.

The report provides both median KLD and 99.9th percentile KLD values. The former reflects quality under typical conditions, while the latter captures extreme cases—those occasional "long-tail" errors that can cause output degradation. The inclusion of 99.9th percentile KLD addresses the non-uniformity of quantization error: certain token positions may produce output deviations far exceeding average levels due to accumulated quantization error—this is precisely the quantitative manifestation of the "occasional hallucination" phenomenon users perceive in practice. This dual-metric presentation is far more representative of actual usage experience than a single average value.

Two Key Techniques: KVarN and Precision Tail

This benchmark introduces two noteworthy technical features:

• KVarN (Variance-Normalized KV-Cache): This is a variance-normalized KV cache quantization scheme proposed by Huawei, implemented in BeeLlama. Its core insight comes from an empirical observation: KV vectors across different attention heads and layers have vastly different numerical distribution ranges (variances). Traditional quantization schemes use the same quantization parameters for all channels—when some channels have much higher variance than others, the effective precision of low-variance channels is severely wasted. KVarN performs variance normalization on each channel before quantization (i.e., dividing by standard deviation), bringing all channels into a similar range before applying uniform quantization. During dequantization, multiplying back by the standard deviation recovers the original values. This preprocessing ensures that quantization error at a fixed bit width is distributed more evenly across channels, preventing a few "outlier" channels from dragging down overall quality—in essence, this is equivalent to adaptively allocating quantization precision to each channel.

• Precision Tail: This strategy retains the KV cache of the most recent X tokens at full (B)F16 precision, only quantizing older historical tokens. Its effectiveness is closely tied to the "recency bias" of Transformer attention mechanisms. Research shows that in most generation scenarios, the model assigns significantly higher attention weights to the most recent tokens compared to distant ones—these recent tokens carry the immediate syntactic and semantic constraints needed for current generation. When KV vectors of these high-attention-weight tokens are quantized with introduced error, the error is amplified through attention weighting and propagated to the output. For more distant historical tokens, even if KV precision is slightly reduced, the impact on final output is relatively limited due to their lower attention weights. A precision tail of 1024 tokens covers approximately 2-3 paragraphs of current conversation, which is sufficient to protect the most critical recent context in most interactive scenarios, trading minimal VRAM overhead for noticeable quality improvement.

The test also covers standard quantization types as well as extended q6_0, q6_1, and even extreme low-bit types from q2_0 to q3_1, providing a complete gradient of options for users with different VRAM budgets.

Qwen Model Results: KVarN's Victory

In the Qwen 3.6 27B (Q5_K_S, 64K context) tests, the most striking finding is: KVarN 6-bit (kvarn6) with a 1024 precision tail surpasses traditional q8_0 in quality.

Specifically, kvarn6 + tail 1024 achieves a median KLD of 0.000879 and a 99.9th percentile KLD of only 0.084629, while its KV cache footprint is just 1744 MiB. In comparison, q8_0 + tail 1024 consumes 2272 MiB with a median KLD of 0.000897. In other words, kvarn6 saves approximately 528 MiB of VRAM while delivering slightly better quality—a rather counterintuitive result.

The technical explanation for this lies in the fact that although traditional q8_0 uses more bits, its quantization precision is distributed "uniformly but not optimally" across channels. KVarN's variance normalization preprocessing ensures that 6-bit quantization precision is more effectively "spent where it matters most"—high-variance channels no longer suffer excessive truncation error after normalization, and low-variance channels no longer waste bits on unnecessary precision headroom.

Even more interesting is the universal effect of precision tail. Comparing the "Qwen Standard-Only" tier without precision tail, q8_0 at 2176 MiB has a KLD of 0.000909, while adding tail 1024 generally improves performance across all tiers. Based on this, the tester identifies "precision tail 1024" as the dominant optimal strategy.

Key tiers from the recommended ladder worth remembering:

• kvarn8 + tail 1024: Best measured quality below BF16, 2256 MiB
• kvarn6 + tail 1024: High-end value pick, 1744 MiB
• kvarn5 + tail 1024: Mid-range best value, 1488 MiB
• kvarn5-kvarn4 hybrid + tail 1024: Balanced default configuration, 1360 MiB
• kvarn3 and below: Only when context must fit into limited VRAM; quality begins to noticeably degrade

Gemma Model Results: Exercise Caution with Aggressive Compression

The Gemma 4 31B (Q5_K_S, 16K context) test results present a strikingly different picture, reminding us not to blindly apply Qwen's findings.

The most significant difference is that Gemma is far more sensitive to quantization than Qwen. Using BF16 as the baseline (KLD near 0), Gemma's median KLD at q8_0 already reaches 0.0371, with the 99.9th percentile KLD soaring to 16.81—absolute values that are orders of magnitude higher than Qwen's. This means KV cache quantization is inherently more "fragile" under the Gemma architecture.

Architectural differences between Gemma and Qwen likely explain their vastly different quantization tolerance. Different models may employ different Group Query Attention (GQA) ratios, RMSNorm placement designs, and attention logits numerical range distributions. If a model's attention score distribution is more "sharp" (i.e., a few tokens hold extremely high attention weights), quantization errors in those critical KV vectors get amplified to a greater degree. Additionally, different training approaches (such as whether long-context fine-tuning was used) also affect the numerical distribution characteristics within the KV cache.

More critically, there is a clear quality cliff. The data shows:

• q5_0 (852.5 MiB) is the "VRAM floor for usable quality," with KLD at 0.061747
• Dropping to q4_0 (697.5 MiB), KLD jumps to 0.134091, marked as "budget tier before the massive cliff"
• Further compression to q3_0 (542.5 MiB) sends KLD soaring to 0.504075
• The extreme q2_0 (387.5 MiB) has KLD of 2.95758, serving only as a "last resort"

The "quality cliff" refers to the phenomenon where a linear decrease in quantization bits does not produce linear quality loss, but instead triggers dramatic quality collapse past a certain critical point. This typically occurs when quantization precision is insufficient to represent critical numerical differences in KV vectors—once quantization resolution falls below a certain threshold, subtle differences in attention weight computation are "smoothed out," causing the model to lose the ability to correctly distinguish between tokens it should attend to and those it should ignore.

Therefore, for Gemma users, q8_0 is the general default for maintaining full prefill speed, q6_0 is the "last tier before the cliff," and q5_0 represents the floor for usable quality. Below this line, quality degradation accelerates dramatically.

Practical Recommendations and Limitations

Synthesizing this benchmark, several practical conclusions emerge:

First, Precision Tail is essentially a free lunch. Retaining the most recent 1024 tokens at BF16 adds only minimal VRAM overhead (approximately 2-4 MiB), yet systematically improves quality—worth enabling by default in any supported scenario.

Second, variance-normalized quantization (KVarN) excels on Qwen, with kvarn6 even surpassing q8_0 at lower VRAM usage—an ideal choice for users who are VRAM-constrained but quality-conscious. This also demonstrates that progress in quantization schemes isn't just about adding more bits, but about more intelligently allocating the representational capacity of limited bits.

Third, different model architectures have vastly different quantization tolerance. Qwen is relatively forgiving and can handle more aggressive compression, while Gemma is far more fragile—it's advisable to conservatively stay at q5_0 or above. The practical implication is that quantization strategies cannot be "one-size-fits-all" and must be validated for specific model architectures.

A note of caution: this test comes from a single Reddit source. While large in scale and transparent in methodology, the performance of techniques like KVarN still requires more independent reproductions for cross-validation. Furthermore, KLD is a distribution-level metric that doesn't perfectly equate to performance on actual downstream tasks (such as code generation or long-form reasoning)—a low KLD guarantees that the statistical properties of the output distribution remain intact, but cannot directly translate to task-level metrics like "X% improvement in code pass rate." Readers should still validate these conclusions against their own use cases.

Nevertheless, this systematic benchmark of 413 configurations provides local LLM deployers with a rare quantization reference map, and its inclusion of emerging KVarN and Precision Tail strategies in the comparison offers considerable forward-looking value.