Unverified50% confidenceFactExact time
For a 70B model using GQA (n_layers=80, n_kv_heads=8, head_dim=128, FP16 cache), the KV Cache size per token is about 320 KB.
1
Sources
50%
Confidence
Long-term
Relevance
10/14/2026
First Seen
Sources
Related Entities
Related Claims
UnverifiedFor a 32-layer, 4096-dimensional model, the KV Cache for processing 2048 tokens occupies approximately 1 GB of VRAM75% similarUnverifiedTaking the LLaMA-3 70B model as an example, with 80 layers, head_dim=128, and BF16 precision, each token occupies approximately 80KB of KV cache.75% similarVerifiedFor a model with 70B parameters, the KV Cache for an 8K context may occupy several additional gigabytes of VRAM74% similarUnverifiedFor 1 million tokens, the KV Cache alone can consume hundreds of GB of VRAM at FP16 precision, far exceeding the 80 GB capacity limit of a single A100/H100.73% similarVerifiedThe VRAM usage of KV Cache grows linearly with sequence length; for a 70-billion-parameter model using GQA, the KV Cache usage can reach tens of GB when processing 100,000 tokens.72% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/1025443API
curl https://kongchang.com/api/v1/knowledge/claims/1025443MCP
get_claim(id=1025443)