KV Cache
KV Cache (Key-Value Cache) is an optimization mechanism used in large language model inference to store the key and value tensors computed during the attention calculation process. By reusing previously computed results, it avoids redundant computation over already-processed tokens, significantly reducing computational overhead during autoregressive generation and improving inference efficiency. KV Cache is widely applied in the deployment and acceleration of Transformer-based models.
Core Facts
The KV cache is a core data structure in Transformer autoregressive inference, and its size is approximately 2 × number of layers × number of attention heads × head dimension × sequence length × precision bytes.
KV Cache reduces the computational complexity of self-attention from O(N²) to O(N) by caching already-computed Key-Value pairs in GPU memory, but memory usage grows linearly with sequence length.
PagedAttention is the core technology of the vLLM framework, which draws on the idea of operating system virtual memory management to store KV Cache in pages.
Timeline (last 90 days)
kv8 means the KV cache uses 8-bit quantization, which can further save VRAM.
The KV cache's validity period varies by scenario: 7 days for chat scenarios and 3 days for agent scenarios.
With the KV cache, the number of tokens processed per second by the same card rises from 60,000 to 1.5 million, a 25-fold difference.
DeepSeek's inference system averages 77,000 input tokens per request, of which 96% hit the KV cache directly.
The KV cache mechanism saves the Key and Value matrices of historical tokens and reuses them in subsequent generation, thereby amortizing the per-unit cost of the input stage.
The output stage generates tokens serially one by one, and the cache cannot skip the computation of generation itself, so vendors have far less room for cost optimization on the output side than on the input side.
AsymLoop uses a single-pass causal encoder to generate layer-specific global KV caches, which remain unchanged throughout the iterative process.
Cache engineering manages the KV cache like memory and is suited to tasks that require multiple rounds of tool calls and long-horizon planning.
The core operations of cache engineering include drop and repos: drop allows the agent to actively exclude selected cache entries from future attention computation; repos adjusts the positional encoding of retained keys.
Traditional methods reduce the KV cache by trimming or summarizing history through context engineering, which breaks the continuity of the trajectory prefix, causing the retained context to require re-prefill, increasing latency and potentially sacrificing task accuracy.
40 more timeline events
All Facts (20)
The KV cache is a core data structure in Transformer autoregressive inference, and its size is approximately 2 × number of layers × number of attention heads × head dimension × sequence length × precision bytes.
90%VerifiedKV Cache reduces the computational complexity of self-attention from O(N²) to O(N) by caching already-computed Key-Value pairs in GPU memory, but memory usage grows linearly with sequence length.
90%VerifiedPagedAttention is the core technology of the vLLM framework, which draws on the idea of operating system virtual memory management to store KV Cache in pages.
90%VerifiedThe VRAM usage and inference latency of MoE architectures are calculated based on the number of activated parameters, not the total parameter count.
90%VerifiedvLLM uses PagedAttention for KV Cache management
80%VerifiedKV Cache is the persistent storage of intermediate states in the Transformer attention mechanism, recording the key and value vectors for each attention head's processed tokens across all layers, thereby skipping redundant forward propagation computations.
80%VerifiedvLLM's PagedAttention draws on the operating system concept of virtual memory paging to manage KV caches, reducing GPU memory fragmentation overhead from approximately 30% to near 0.
80%Verified以LLaMA-3 70B（80层、64头、每头128维）为例，单个token的KV Cache约占2.6MB（FP16精度），32K上下文的KV Cache总量约达83GB
80%VerifiedKV Cache是Transformer架构的核心推理优化机制，其显存占用量会随序列长度和并发批次线性增长
80%VerifiedPagedAttention技术借鉴操作系统虚拟内存的分页思想，将KV Cache切分为固定大小的物理块动态分配
80%VerifiedQuantized KV caches (such as storing KV vectors in FP8 or INT4) can compress the cache size to 1/2 or even 1/4 of the original.
80%VerifiedThe KV Cache mechanism can cache repeated prefix content, and the more concise the system prompt, the higher the cache hit rate.
80%VerifiedFor a model with 70B parameters, the KV Cache for an 8K context may occupy several additional gigabytes of VRAM
80%VerifiedThe longer the context, the more the KV cache grows linearly, making memory bandwidth and capacity the main bottleneck for inference throughput.
75%VerifiedThe cost of persistent KV caching is a significant increase in storage space, along with the engineering challenge of efficiently locating and loading relevant segments from a massive cache.
75%VerifiedThe KV cache grows linearly with context length, increasing the memory read volume per step in the decoding phase, which is the fundamental reason decoding speed decreases as context grows.
75%VerifiedKVCache is a mechanism in the Transformer architecture that caches the Key and Value vectors of all previous tokens to avoid redundant computation, with its memory footprint growing linearly with context length.
75%VerifiedCompared to the previous-generation V4 Flash, V4.1 Flash reduces the demand for high-bandwidth memory (HBM) to one-quarter and the demand for SSD persistent storage to one-eighth.
75%VerifiedIn the KV Cache mechanism, reusing cached KV values is a cache hit, while context changes that invalidate the cache and require recomputation are a cache miss.
75%VerifiedThe KV Cache (Key-Value Cache) strategy stores already-computed attention key-value pairs in GPU memory for reuse, which consumes a large amount of GPU memory resources.
75%