Unverified50% confidenceConceptExact time
The computational cost of long-context attention grows quadratically with sequence length, and Key and Value vectors must be cached for each token (KV Cache), which at a 1M context can reach hundreds of GB.
1
Sources
50%
Confidence
Long-term
Relevance
10/16/2026
First Seen
Sources
Related Entities
Related Claims
VerifiedWhen the context length is extended to one million tokens, the KV Cache itself can occupy tens of gigabytes of VRAM.80% similarUnverifiedIn large model API call scenarios, cache hit rate refers to the reuse ratio of the KV Cache (key-value cache), where the model can directly read the attention key-value pairs of already-processed tokens from the cache.79% similarVerifiedThe VRAM consumption of the KV cache is proportional to context length, number of model layers, and number of attention heads; under an 8K context, a typical 27B model may consume an additional 2–4 GB of VRAM.78% similarVerifiedFor a typical 7B model with 32 layers and 32 attention heads, storing the KV Cache for each token requires approximately 0.5 MB of VRAM, so generating 1,000 tokens requires an additional ~500 MB of VRAM77% similarVerifiedThe VRAM usage of KV Cache grows linearly with sequence length; for a 70-billion-parameter model using GQA, the KV Cache usage can reach tens of GB when processing 100,000 tokens.77% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/1044059API
curl https://kongchang.com/api/v1/knowledge/claims/1044059MCP
get_claim(id=1044059)