Unverified50% confidenceFactExact time
Every 1,000 in-flight tokens require about 0.33 GB of VRAM in the KV cache.
1
Sources
50%
Confidence
Long-term
Relevance
10/13/2026
First Seen
Sources
Related Entities
Related Claims
UnverifiedFor 1 million tokens, the KV Cache alone can consume hundreds of GB of VRAM at FP16 precision, far exceeding the 80 GB capacity limit of a single A100/H100.79% similarVerifiedFor a typical 7B model with 32 layers and 32 attention heads, storing the KV Cache for each token requires approximately 0.5 MB of VRAM, so generating 1,000 tokens requires an additional ~500 MB of VRAM78% similarUnverifiedIn the replaced model, the tied embedding/lm_head matrix remains bf16, so each token actually reads about 1.263 GB of weights (plus about 0.06 GB of KV cache); calculated by actual bytes moved, 25.5 tok/s corresponds to about 34 GB/s, roughly 10% of measured bandwidth.76% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/1019923API
curl https://kongchang.com/api/v1/knowledge/claims/1019923MCP
get_claim(id=1019923)