Unverified50% confidenceFactExact time
In disaggregated inference, the KV cache generated by prefill nodes must be transferred to decode nodes in real time, and for long-context or large-batch requests a single transfer may reach several GB.
1
Sources
50%
Confidence
Long-term
Relevance
10/11/2026
First Seen
Sources
Related Entities
Related Claims
UnverifiedFor long-context requests, the KV cache data volume can reach several GB72% similarVerifiedThe disk caching mechanism persists already-computed KV Cache to storage media; when a subsequent request's prefix hits the cache, it can be reused directly, skipping redundant forward propagation.71% similarUnverifiedDisaggregated inference decouples the compute-intensive prefill stage from the memory-bandwidth-intensive decode stage, allocating them to separate hardware resource pools best suited to each for independent scaling.71% similarUnverified自回归解码阶段每生成一个新token都需读取完整KV Cache，使解码阶段几乎总是受限于内存带宽而非计算能力71% similarUnverifiedllama.cpp Adaptive KV Streaming changes data movement from being managed automatically by the system to being actively scheduled by the inference program; the full KV is stored in memory, only a portion resides permanently in VRAM, and a circular buffer prefetches the KV needed by subsequent layers.70% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/995755API
curl https://kongchang.com/api/v1/knowledge/claims/995755MCP
get_claim(id=995755)