Unverified50% confidenceFactExact time
In traditional Transformer inference, the KV Cache for each request requires pre-allocation of contiguous GPU memory; because sequence length is unpredictable, allocation is often done at maximum length, leading to memory fragmentation and waste, with actual utilization sometimes below 50%.
1
Sources
50%
Confidence
Long-term
Relevance
9/10/2026
First Seen
Sources
Related Entities
Related Claims
UnverifiedThe KV Cache of Transformers grows quadratically with sequence length, leading to non-linear GPU memory consumption during long-context inference.73% similarUnverifiedKV Cache往往占据推理总显存的30%至70%，是制约大模型推理吞吐量和上下文长度的主要瓶颈72% similarVerifiedKV Cache reduces the time complexity of Transformer incremental generation from O(L²) to O(L).71% similarUnverifiedThe prefill stage of Transformer inference is primarily bottlenecked by compute (FLOPS), while the decode stage is primarily bottlenecked by memory bandwidth.69% similarUnverifiedThe computational cost of one recurrent depth iteration is roughly equivalent to a forward pass through one Transformer layer, but without the need to perform softmax sampling over the vocabulary and token decoding, so the overhead may be orders of magnitude lower for the same amount of thinking.68% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/888204API
curl https://kongchang.com/api/v1/knowledge/claims/888204MCP
get_claim(id=888204)