Unverified50% confidenceFactExact time
When generating each token, a Transformer model needs to compute the Key and Value vectors of all previous tokens, and these tensors are collectively called the KV cache.
1
Sources
50%
Confidence
Long-term
Relevance
10/14/2026
First Seen
Sources
Related Entities
Related Claims
UnverifiedDuring Transformer model inference, input tokens can be processed in parallel (prefill stage), while output tokens must be generated one by one autoregressively (decode stage), causing the unit price of output tokens to typically be higher than that of input tokens.78% similarUnverifiedThe context cache (KV Cache) is the memory area that stores the Key and Value vectors of previous tokens during inference of Transformer-type models, and its size grows linearly with context length, batch size, and the number of model layers.75% similarPartially VerifiedThe KV cache is a core data structure in Transformer autoregressive inference, and its size is approximately 2 × number of layers × number of attention heads × head dimension × sequence length × precision bytes.74% similarUnverifiedIn standard Transformer inference, generating each new token requires accessing the key-value vectors of all historical tokens, and the memory footprint of the KV cache grows linearly with sequence length.74% similarVerifiedKV Cache是Transformer架构的核心推理优化机制，其显存占用量会随序列长度和并发批次线性增长72% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/1031674API
curl https://kongchang.com/api/v1/knowledge/claims/1031674MCP
get_claim(id=1031674)