Partially Verified65% confidenceFactExact time
Hugging Face calculates that the KV cache under BF16 is about 250 KB per token, and a full 1-million-token context requires about 266 GB.
3
Sources
65%
Confidence
Long-term
Relevance
10/15/2026
First Seen
Sources
Related Entities
Related Claims
UnverifiedIn the replaced model, the tied embedding/lm_head matrix remains bf16, so each token actually reads about 1.263 GB of weights (plus about 0.06 GB of KV cache); calculated by actual bytes moved, 25.5 tok/s corresponds to about 34 GB/s, roughly 10% of measured bandwidth.75% similarVerifiedThe VRAM consumption of the KV cache is proportional to context length, number of model layers, and number of attention heads; under an 8K context, a typical 27B model may consume an additional 2–4 GB of VRAM.74% similarUnverifiedThe KV cache size can be expressed with the formula cache = 2 (K and V) × layers × KV heads × head dim × tokens × 2 bytes, where 2 bytes corresponds to FP16, doubling when using FP32, and halving when using INT8 quantization.74% similarVerifiedFor a typical 7B model with 32 layers and 32 attention heads, storing the KV Cache for each token requires approximately 0.5 MB of VRAM, so generating 1,000 tokens requires an additional ~500 MB of VRAM74% similarVerifiedTaking LLaMA-2 70B as an example, when processing a 4096-token context with a batch size of 32, the KV Cache can consume over 80 GB of memory.74% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/1041160API
curl https://kongchang.com/api/v1/knowledge/claims/1041160MCP
get_claim(id=1041160)