Unverified50% confidenceFactExact time
In tests on Qwen3.8 27B, the 8-bit KV cache of llama.cpp and ExLlamaV3 (q8_0 and EXL3 8-bit) produced KL divergence changes within ±0.00012 across all tested weight quantization versions, falling within the range of numerical noise.
1
Sources
50%
Confidence
Long-term
Relevance
11/7/2026
First Seen
Sources
Related Entities
Related Claims
VerifiedWhen using FP16 precision, the KV Cache for LLaMA-2-7B occupies approximately 0.5 MB per token.72% similarUnverifiedThe Q4_K_M/GGUF version of Qwen 27B decodes at approximately 10.7 tok/s on the same machine, roughly 1/3.8 of the MLX version.71% similarUnverifiedAfter adopting GQA, Llama 3.3 70B has a KV Cache of approximately 320 KB per token.71% similarUnverifiedIn the Qwen3.8 27B vLLM tests, using the Marlin kernel to keep quantized weights while keeping activations in bf16 lowered NVIDIA NVFP4's KLD from 0.0256 to 0.0117, a reduction of more than half.71% similarVerifiedTaking LLaMA-3 70B (80 layers, 64 heads, 128 dimensions per head) as an example, the KV Cache for a single token occupies approximately 2.6 MB (FP16 precision), and the total KV Cache for a 32K context reaches approximately 83 GB.68% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/1146816API
curl https://kongchang.com/api/v1/knowledge/claims/1146816MCP
get_claim(id=1146816)