PagedAttention
A core technical innovation in vLLM that manages KV Cache in GPU memory by analogy with the virtual memory paging mechanism of operating systems, substantially improving GPU memory utilization and concurrent throughput.
Core Facts
vLLM is an open-source large language model inference engine project built on top of PyTorch.
PagedAttention is the core technology of the vLLM framework, which draws on the idea of operating system virtual memory management to store KV Cache in pages.
vLLM achieves efficient KV Cache management through PagedAttention technology, improving GPU memory utilization several times over.
Timeline (last 90 days)
Methods for optimizing the KV cache include grouped-query attention (GQA), paged attention (such as vLLM's PagedAttention), and sliding window attention.
PagedAttention reduces VRAM fragmentation through block-level on-demand allocation and allows different sequences to share the same physical block, saving VRAM in scenarios such as beam search, parallel sampling, and prefix sharing.
The core idea of PagedAttention borrows from the paging mechanism of operating system virtual memory, so that blocks need not be contiguous in physical VRAM.
vLLM's PagedAttention mechanism divides the KV cache into blocks of a fixed number of tokens (usually 16), and records the mapping from logical positions to physical blocks through a block table.
Inference-side engineering optimizations such as quantization, speculative decoding, and PagedAttention are becoming the core means by which vendors widen the cost gap.
PagedAttention is the foundation of Continuous Batching, where new requests can be inserted into a running batch at any time without waiting for the entire batch to finish.
PagedAttention cuts the KV Cache into fixed-size blocks, allocated on demand and stored non-contiguously, raising VRAM utilization from the traditional 20%–40% to over 90%.
PagedAttention causes KV cache memory waste to occur only in the last unfilled block of a request, and vLLM reports waste of less than 4%.
PagedAttention divides KV memory into fixed-size blocks, each request sees contiguous logical blocks mapped through a block table to physical blocks at arbitrary locations, physical blocks are allocated on demand, and the attention kernel is rewritten to follow the block table.
vLLM's core innovation is the PagedAttention memory management mechanism, which can improve GPU memory utilization of the KV Cache from the conventional 40–60% to over 90%.
40 more timeline events
All Facts (20)
vLLM is an open-source large language model inference engine project built on top of PyTorch.
90%VerifiedPagedAttention is the core technology of the vLLM framework, which draws on the idea of operating system virtual memory management to store KV Cache in pages.
90%VerifiedvLLM achieves efficient KV Cache management through PagedAttention technology, improving GPU memory utilization several times over.
90%VerifiedvLLM是由UC Berkeley开发的高吞吐量大模型推理引擎，核心创新是PagedAttention技术
90%VerifiedPagedAttention divides the KV cache into fixed-size pages allocated on demand, and can improve VRAM utilization by 2–4×.
85%VerifiedPagedAttention improves KV Cache memory utilization from below 40% to close to 100%.
85%VerifiedPaged Attention (PagedAttention) is the core technology of vLLM, managing KV cache in blocks, while Flash Attention optimizes the memory access pattern of attention computation.
80%VerifiedvLLM's PagedAttention draws on the operating system concept of virtual memory paging to manage KV caches, reducing GPU memory fragmentation overhead from approximately 30% to near 0.
80%VerifiedThe PagedAttention mechanism proposed by vLLM improves its throughput by 2–24× compared to HuggingFace Transformers.
80%VerifiedModern inference frameworks such as vLLM and TensorRT-LLM improve GPU utilization through techniques such as Continuous Batching and PagedAttention.
80%VerifiedPagedAttention技术借鉴操作系统虚拟内存的分页思想，将KV Cache切分为固定大小的物理块动态分配
80%VerifiedKV Cache avoids redundant computation by caching already-computed Key and Value matrices, and advanced techniques such as PagedAttention further optimize cache memory management efficiency.
80%VerifiedvLLM uses PagedAttention for KV Cache management
80%VerifiedvLLM employs Continuous Batching, refining the scheduling granularity to each generation step, filling slots as soon as they become available.
80%VerifiedvLLM is one of the most mainstream open-source inference serving frameworks, implementing efficient KV cache management through its PagedAttention technology.
75%VerifiedThe KV Cache (Key-Value Cache) strategy stores already-computed attention key-value pairs in GPU memory for reuse, which consumes a large amount of GPU memory resources.
75%VerifiedInference engines such as vLLM and TensorRT-LLM improve GPU throughput through techniques like PagedAttention and continuous batching.
65%VerifiedFlashAttention通过重新设计注意力计算的内存访问模式，将显存占用从与序列长度平方成正比降低为线性关系
65%VerifiedPagedAttention splits the KV Cache into fixed-size physical blocks mapped via a block table, and can improve GPU utilization from below 40% to above 90%.
65%UnverifiedvLLM introduced PagedAttention as an optimization technique
90%