KV Cache
KV Cache (Key-Value Cache) is an optimization mechanism used during Transformer model inference to store the key and value vectors corresponding to each token in the attention computation. During autoregressive generation, the model reuses cached key-value pairs from previous tokens rather than recomputing them at each new generation step, significantly reducing redundant computation and improving inference efficiency. The mechanism trades increased GPU memory usage for faster inference speed and is a core optimization technique in large language model deployment.
Core Facts
KV Cache reduces the computational complexity of self-attention from O(N²) to O(N) by caching already-computed Key-Value pairs in GPU memory, but memory usage grows linearly with sequence length.
The VRAM occupied by KV Cache grows linearly with sequence length.
PagedAttention is the core technology of the vLLM framework, which draws on the idea of operating system virtual memory management to store KV Cache in pages.
Timeline (last 90 days)
The context cache (KV Cache) is the memory area that stores the Key and Value vectors of previous tokens during inference of Transformer-type models, and its size grows linearly with context length, batch size, and the number of model layers.
Evicting the KV entries of low-value tokens has become a key means of reducing the cost of long-context inference.
In traditional KV cache research, rules and implementations are tightly bound, so when method A outperforms method B, it is impossible to tell whether the eviction logic is smarter or the kernel is more hardware-friendly.
The longer the context, the more the KV cache grows linearly, making memory bandwidth and capacity the main bottleneck for inference throughput.
The KV cache is a core data structure in Transformer autoregressive inference, and its size is approximately 2 × number of layers × number of attention heads × head dimension × sequence length × precision bytes.
The hardware requirements of large model inference differ fundamentally from those of training: training requires extremely large VRAM, extremely high bandwidth, and strong interconnect, while inference's bottleneck lies more in weight loading and KV Cache storage.
Streaming loading schemes can be used in combination with techniques such as quantization, distillation, and KV Cache optimization; quantization shrinks the size of a single expert, while streaming loading solves the total-volume problem.
DeepSeek V4.1 Flash lets the read and write paths share the same KV state: the read phase establishes and updates this memory, and the generation phase directly reuses it, rather than starting another cache.
Agentic serving workloads have three notable characteristics: high interdependence between requests, unpredictable output length for a single request, and the need for the KV Cache to be reused or quickly switched among multiple calls.
Atomic Agent chose to attach a rejection message ('you are responsible for planning, the Worker is responsible for writing') to the tool call's returned result rather than dynamically removing the write tool, in order to protect the local planning model's KV cache and avoid cache invalidation caused by a change in the prompt prefix.
40 more timeline events
All Facts (20)
KV Cache reduces the computational complexity of self-attention from O(N²) to O(N) by caching already-computed Key-Value pairs in GPU memory, but memory usage grows linearly with sequence length.
90%VerifiedThe VRAM occupied by KV Cache grows linearly with sequence length.
90%VerifiedPagedAttention is the core technology of the vLLM framework, which draws on the idea of operating system virtual memory management to store KV Cache in pages.
90%VerifiedKV Cache reduces inference complexity from O(n²) to O(n) by caching the already-computed Key/Value matrices.
90%VerifiedPagedAttention divides the KV cache into fixed-size pages allocated on demand, and can improve VRAM utilization by 2–4×.
85%VerifiedPagedAttention improves KV Cache memory utilization from below 40% to close to 100%.
85%VerifiedKV Cache is the persistent storage of intermediate states in the Transformer attention mechanism, recording the key and value vectors for each attention head's processed tokens across all layers, thereby skipping redundant forward propagation computations.
80%VerifiedPaged Attention (PagedAttention) is the core technology of vLLM, managing KV cache in blocks, while Flash Attention optimizes the memory access pattern of attention computation.
80%VerifiedGrouped Query Attention (GQA) and Multi-Query Attention (MQA) reduce KV cache memory usage during inference by having multiple query heads share the same group of key/value heads, and are standard optimizations in models such as Llama 3 and Mistral.
80%VerifiedTaking LLaMA-2 70B as an example, when processing a 4096-token context with a batch size of 32, the KV Cache can consume over 80 GB of memory.
80%VerifiedLLaMA-2-70B's KV Cache for a single request at a context length of 128K can occupy more than 160 GB of VRAM.
80%VerifiedThe VRAM usage of KV Cache grows linearly with sequence length; for a 70-billion-parameter model using GQA, the KV Cache usage can reach tens of GB when processing 100,000 tokens.
80%Verified大型语言模型处理长上下文时，注意力机制的计算复杂度与序列长度的平方成正比
80%Verified以LLaMA-3 70B（80层、64头、每头128维）为例，单个token的KV Cache约占2.6MB（FP16精度），32K上下文的KV Cache总量约达83GB
80%VerifiedTransformer架构的自注意力机制在处理长序列时面临二次方复杂度的计算开销，上下文长度翻倍则计算量增长四倍
80%VerifiedKV Cache是Transformer架构的核心推理优化机制，其显存占用量会随序列长度和并发批次线性增长
80%VerifiedPagedAttention技术借鉴操作系统虚拟内存的分页思想，将KV Cache切分为固定大小的物理块动态分配
80%VerifiedKV Cache avoids redundant computation by caching already-computed Key and Value matrices, and advanced techniques such as PagedAttention further optimize cache memory management efficiency.
80%VerifiedvLLM uses PagedAttention for KV Cache management
80%VerifiedThe KV Cache mechanism can cache repeated prefix content, and the more concise the system prompt, the higher the cache hit rate.
80%