KV Cache
KV Cache (Key-Value Cache) is an optimization mechanism used in large language model inference to store the key and value tensors computed during the attention calculation process. By reusing previously computed results, it avoids redundant computation over already-processed tokens, significantly reducing computational overhead during autoregressive generation and improving inference efficiency. KV Cache is widely applied in the deployment and acceleration of Transformer-based models.
Core Facts
KV Cache reduces the computational complexity of self-attention from O(N²) to O(N) by caching already-computed Key-Value pairs in GPU memory, but memory usage grows linearly with sequence length.
PagedAttention is the core technology of the vLLM framework, which draws on the idea of operating system virtual memory management to store KV Cache in pages.
The VRAM usage and inference latency of MoE architectures are calculated based on the number of activated parameters, not the total parameter count.
Timeline (last 90 days)
Static system prompts enjoy a 90% KV cache discount, but dynamic chain-of-thought traces are stripped between turns or regenerated at each step, so prefix caching cannot amortize test-time compute bursts across multi-turn agent sessions.
KV cache reuse can reduce redundant computation, and operator fusion and quantization (such as INT8/FP16) can lower the per-inference overhead.
When two requests hit the same inference engine simultaneously, the engine attempts to reuse or extend the same KV cache, but the differing conversation histories of the two sessions cause conflicts in cache indices and sequence positions, resulting in garbled output.
When multiple Agent sessions running simultaneously on a single 3090 hit the same Qwen inference engine, concurrent requests corrupt the KV cache in use, causing a sharp drop in generation quality.
Transformer's KV cache size grows linearly with sequence length, whereas Mamba2's recurrent state cache size is independent of context length and depends only on the state dimension and the number of layers.
If sliding-window memory slides once per turn, it causes the tokens at the beginning of the history to change, invalidating the entire KV cache of the history; the fix is batch eviction (for example, growing from 20 to 40 and then cutting back to 20 all at once), so that the cache keeps hitting between two trims.
The KV cache stores the attention key-value pairs of all previous tokens during an Agent's multi-turn interactions to avoid recomputation, but its VRAM usage grows linearly or even near-linearly with context length, so long-context Agents easily hit the VRAM limit.
The inference engine reuses shared prefixes via the KV cache, letting new tokens skip repeated prefill computation, but the tokenization step has no similar reuse shortcut, and the CPU frontend of the serving engine must reconvert the entire request into token IDs from the first character every time.
TTFT (time to first token) consists of two segments: frontend request preprocessing (tokenization, prompt concatenation) and backend prefill computation, and the KV cache only optimizes the latter.
KV cache size grows linearly with context length, and attention computation grows quadratically with context length.
40 more timeline events
All Facts (20)
KV Cache reduces the computational complexity of self-attention from O(N²) to O(N) by caching already-computed Key-Value pairs in GPU memory, but memory usage grows linearly with sequence length.
90%VerifiedPagedAttention is the core technology of the vLLM framework, which draws on the idea of operating system virtual memory management to store KV Cache in pages.
90%VerifiedThe VRAM usage and inference latency of MoE architectures are calculated based on the number of activated parameters, not the total parameter count.
90%VerifiedvLLM uses PagedAttention for KV Cache management
80%VerifiedKV Cache is the persistent storage of intermediate states in the Transformer attention mechanism, recording the key and value vectors for each attention head's processed tokens across all layers, thereby skipping redundant forward propagation computations.
80%VerifiedvLLM's PagedAttention draws on the operating system concept of virtual memory paging to manage KV caches, reducing GPU memory fragmentation overhead from approximately 30% to near 0.
80%Verified以LLaMA-3 70B（80层、64头、每头128维）为例，单个token的KV Cache约占2.6MB（FP16精度），32K上下文的KV Cache总量约达83GB
80%VerifiedKV Cache是Transformer架构的核心推理优化机制，其显存占用量会随序列长度和并发批次线性增长
80%VerifiedPagedAttention技术借鉴操作系统虚拟内存的分页思想，将KV Cache切分为固定大小的物理块动态分配
80%VerifiedQuantized KV caches (such as storing KV vectors in FP8 or INT4) can compress the cache size to 1/2 or even 1/4 of the original.
80%VerifiedThe KV Cache mechanism can cache repeated prefix content, and the more concise the system prompt, the higher the cache hit rate.
80%VerifiedFor a model with 70B parameters, the KV Cache for an 8K context may occupy several additional gigabytes of VRAM
80%VerifiedThe KV cache grows linearly with context length, increasing the memory read volume per step in the decoding phase, which is the fundamental reason decoding speed decreases as context grows.
75%VerifiedCompared to the previous-generation V4 Flash, V4.1 Flash reduces the demand for high-bandwidth memory (HBM) to one-quarter and the demand for SSD persistent storage to one-eighth.
75%VerifiedIn the KV Cache mechanism, reusing cached KV values is a cache hit, while context changes that invalidate the cache and require recomputation are a cache miss.
75%VerifiedThe KV Cache (Key-Value Cache) strategy stores already-computed attention key-value pairs in GPU memory for reuse, which consumes a large amount of GPU memory resources.
75%VerifiedThe KV cache VRAM usage of ultra-long context windows grows linearly with context length, causing inference latency and costs to rise significantly.
75%VerifiedThe KV Cache (key-value cache) mechanism accelerates inference by caching the Key and Value vectors generated by the self-attention layer to avoid redundant computation, but the longer the context, the more vectors must reside in VRAM, causing memory pressure to grow in a near-linear fashion.
75%VerifiedKV缓存将已计算的Key-Value矩阵存储在显存中实现增量计算，代价是显存占用随序列长度线性增长
75%VerifiedDuring the autoregressive generation phase, generating each new token requires recalculating attention over all historical tokens, with computational complexity growing as O(n²) with sequence length; KV Cache can reduce this to O(n) incremental operations.
75%