Verified90% confidenceFactExact time
The KV cache is a core data structure in Transformer autoregressive inference, and its size is approximately 2 × number of layers × number of attention heads × head dimension × sequence length × precision bytes.
6
Sources
90%
Confidence
Long-term
Relevance
10/13/2026
First Seen
Sources
Related Entities
Related Claims
UnverifiedKV cache size is proportional to the number of model layers, the number of attention heads, sequence length, and per-head dimension, and is the primary source of VRAM usage during large model inference.80% similarVerifiedKV Cache是Transformer架构的核心推理优化机制，其显存占用量会随序列长度和并发批次线性增长79% similarUnverifiedThe KV cache, as the carrier of the inference state of Transformer-type models, naturally mixes information from different sources and lacks a mechanism for isolation by relationship.79% similarUnverifiedWhen generating each token, a Transformer model needs to compute the Key and Value vectors of all previous tokens, and these tensors are collectively called the KV cache.74% similarUnverifiedPrefix cache reuse causes KV values to be computed in a different batch than a fresh prefill, so the KV values from a cache hit may differ at the floating-point level from those of a fresh prefill, and the difference propagates through subsequent tokens.73% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/1020971API
curl https://kongchang.com/api/v1/knowledge/claims/1020971MCP
get_claim(id=1020971)