PagedAttention Explained: GPU Memory Optimization and Model Routing Techniques

Introduction: The Memory Engineering Behind AI Inference

As large language models (LLMs) continue to grow in scale, efficiently utilizing GPU memory has become one of the most critical challenges in AI infrastructure. Recently, a technical researcher (LinkedIn: gustavkeller) shared a comprehensive whitepaper on Reddit covering GPU memory mechanisms, PagedAttention, and model routing, sparking widespread discussion across the ML and AI Infra communities.

While the author modestly noted that "the grammar might not be perfect," the technical value of this whitepaper is undeniable. This article will explore these three core topics, outlining the key engineering practices and optimization strategies for memory management in modern LLM inference systems.

GPU Memory Mechanisms: The Hidden Bottleneck of LLM Inference

Three Major Sources of Memory Consumption

In LLM inference scenarios, GPU memory consumption primarily comes from three sources: model weights, activations, and KV Cache (Key-Value Cache). Model weights represent a fixed overhead, while KV Cache grows dramatically with increasing sequence length and concurrent request volume.

Take a typical Transformer model as an example: for every token processed, the model needs to cache the corresponding Key and Value vectors for each layer. When context windows reach tens of thousands of tokens while serving hundreds or thousands of concurrent requests, KV Cache memory usage often exceeds the model weights themselves.

KV Cache is a core optimization mechanism in the Transformer architecture during autoregressive generation. In standard self-attention computation, generating each new token requires recomputing Key and Value vectors for all historical tokens, with O(n²) computational complexity. KV Cache stores previously computed Key and Value matrices, so each generation step only needs to compute the current token's Query and perform dot-product attention with the cached Keys, reducing incremental computation to O(n). However, this optimization is a classic space-for-time tradeoff—for a model with L layers, hidden dimension d, and h attention heads, a single sequence of length n requires 2×L×n×h×d_head×sizeof(dtype) bytes of KV Cache memory. For LLaMA-70B, with 80 layers, 64 attention heads, 128 dimensions per head, and FP16 precision, the KV Cache for a single sequence of length 4096 requires approximately 5GB of memory—explaining why KV Cache management has become the central challenge of inference engineering.

Two Major Flaws in Traditional Memory Allocation

Traditional inference frameworks typically pre-allocate a contiguous memory region for each request's KV Cache, sized according to the maximum possible sequence length. This approach introduces two serious problems:

• Internal fragmentation: Actual sequence lengths are far shorter than the reserved space, resulting in significant memory waste.
• External fragmentation: Allocation and deallocation of contiguous memory blocks leaves unusable gaps in memory.

Research shows that under this static allocation approach, actual memory utilization can be as low as 20%-40%, severely limiting the system's concurrent capacity. This is precisely the core problem that PagedAttention aims to solve.

PagedAttention: Borrowing the OS Paging Paradigm

Core Design Philosophy and Working Mechanism

PagedAttention draws inspiration from virtual memory and paging mechanisms in operating systems. It divides KV Cache into fixed-size "blocks," each capable of storing key-value pairs for a fixed number of tokens. These blocks don't need to be stored contiguously in physical memory—instead, a "block table" maintains the mapping between logical sequences and physical blocks.

To appreciate the elegance of PagedAttention's design, it helps to revisit the OS paging mechanism it borrows from—one of the most classic abstractions in computer science. In operating systems, physical memory is divided into fixed-size page frames (typically 4KB), and a process's virtual address space is mapped to non-contiguous physical page frames via a Page Table. This design solved the external fragmentation problem caused by early contiguous memory allocation, freeing processes from needing to know the actual physical memory layout. Additionally, operating systems introduced Copy-on-Write (COW) technology—when multiple processes share the same physical page, the page is only actually copied when a process attempts to write to it, saving memory and reducing overhead for operations like fork. PagedAttention migrates these OS design patterns, proven over decades, to GPU memory management—demonstrating the engineering wisdom of cross-domain knowledge transfer in system design.

This design brings a fundamental change: memory no longer needs to reserve large contiguous blocks for each request. Instead, it allocates on demand and grows dynamically. When a sequence generates new tokens, the system simply allocates new physical blocks and updates the mapping.

Three Core Advantages of PagedAttention

With the paging mechanism, inference systems gain the following benefits:

1. Near-complete elimination of memory fragmentation: Since allocation happens in fixed-size blocks, internal fragmentation is limited to the last unfilled block, while external fragmentation is completely eliminated.
2. Memory sharing support: In parallel sampling, Beam Search, and similar scenarios, multiple sequences can share the same prefix blocks through Copy-on-Write mechanisms, avoiding redundant storage.
3. Significantly improved inference throughput: Higher memory utilization means the system can accommodate more concurrent requests, dramatically boosting overall throughput.

Regarding the second point, Beam Search is a search algorithm widely used in machine translation, text generation, and other tasks that retains the top-k most probable candidate sequences (beams) at each step, ultimately selecting the best result. In traditional implementations, k beams each maintain independent KV Cache copies, but they actually share identical prefix history before diverging. Similarly, in temperature sampling scenarios, the same prompt may need to generate multiple candidate responses whose KV Caches are identical during the prompt phase. PagedAttention uses reference counting and Copy-on-Write mechanisms to allow multiple sequences to share the same physical blocks, only allocating new blocks when a sequence diverges. This design can save approximately 55% of KV Cache memory overhead in a typical beam_width=4 configuration.

This technology is the key foundation enabling high-performance inference engines like vLLM to achieve multi-fold throughput improvements. vLLM is a high-performance LLM inference engine open-sourced by UC Berkeley's Sky Computing Lab in 2023, with PagedAttention as its core innovation. In benchmarks at launch, vLLM achieved 2-4x throughput improvement over the then-mainstream HuggingFace Transformers inference approach. By combining Continuous Batching with PagedAttention, vLLM can dynamically insert new requests into executing batches, avoiding the resource waste in traditional static batching where short sequences wait for long sequences to complete. Comparable solutions include NVIDIA's TensorRT-LLM (focused on compilation optimization and quantization acceleration), DeepSpeed-FastGen, and the recently emerging SGLang, each with different focuses but all incorporating similar paging concepts for memory management.

It has pushed AI inference from coarse resource occupation toward fine-grained memory engineering.

Model Routing: Intelligent Scheduling for Multi-Model Services

From Single Models to Multi-Model Orchestration

As AI applications grow increasingly complex, a single model often struggles to achieve optimal balance among cost, latency, and quality. Model Routing emerged to address this challenge—its core idea is to dynamically dispatch requests to the most suitable model based on request characteristics.

For example, a simple Q&A request can be handled by a smaller, faster-responding model, while tasks involving complex reasoning are routed to more powerful but costlier models. This strategy significantly reduces average inference costs while maintaining service quality.

Model routing has multiple mature implementations in industry. OpenAI's GPT-4 system is speculated to internally use a Mixture of Experts (MoE) architecture, which is essentially token-level routing. At the service level, startups like Martian and Unify provide request-level model routing services that automatically select the most suitable LLM API based on input semantic complexity, domain characteristics, and other factors. In academia, research like RouteLLM (proposed by the LMSys team) trains dedicated routing models to optimally distribute between strong and weak models, reducing costs by 40-60% while maintaining over 95% quality.

Key Dimensions in Routing Strategy Design

Effective model routing typically requires consideration of the following factors:

• Task complexity assessment: Using lightweight classifiers or heuristic rules to judge request difficulty. Typical routing classifiers can use lightweight models like BERT to complete difficulty assessment in a few milliseconds, with computational overhead that is negligible compared to the cost savings from routing decisions.
• Cost and latency constraints: Selecting the most cost-effective model while meeting SLA requirements.
• Load balancing: Preventing excessive concentration of requests on a single model instance, which could create memory and compute bottlenecks.

Model routing complements the memory management techniques discussed earlier: memory optimization technologies like PagedAttention enhance individual model serving capacity, while model routing achieves optimal resource allocation at the system level. Together, they form the technical backbone of modern AI inference infrastructure.

Conclusion: The Engineering Value of AI Inference Infrastructure

The three topics covered in this whitepaper—GPU memory mechanisms, PagedAttention, and model routing—effectively outline the complete optimization landscape for LLM inference systems: from low-level physical memory management, to mid-level KV Cache paging algorithms, to high-level model scheduling strategies.

For practitioners in ML engineering or AI infrastructure, understanding these mechanisms not only helps in selecting appropriate inference frameworks (such as vLLM, TensorRT-LLM, etc.) but also enables critical architectural decisions in cost-sensitive production environments. As large models move toward large-scale deployment, AI Infra engineering capability is becoming one of the core factors determining product competitiveness.

Note: This article is based on a technical whitepaper shared in the Reddit community. The original author is not a professional writer; for in-depth discussion, you can reach them via LinkedIn (gustavkeller).