Deep Dive into LLM Inference: A Complete Guide from KV Cache to Prefill-Decode Disaggregation

Large model inference and serving is one of the most active and rapidly evolving areas in AI infrastructure today. Peking University's Weiming Supercomputing Team, in collaboration with LCPU, hosted an AI Infra Seminar focused on "Inference & LLM Serving," systematically covering the full technical stack from KV Cache fundamentals to distributed serving strategies. This article distills the core design principles and key engineering trade-offs from that seminar.

The Two-Phase Nature of Inference

An autoregressive model's inference request passes through two fundamentally different phases: Prefill and Decode. The Prefill phase processes the entire prompt at once as a standard GEMM (matrix multiplication), achieving high arithmetic intensity — it is compute-bound. The Decode phase, by contrast, generates one token at a time, resembling a GEMV (matrix-vector multiplication), with extremely low arithmetic intensity — it is memory-bound.

This "Prefill is compute-bound, Decode is memory-bound" conclusion has been widely cited, but the seminar stressed the need to think critically about it — the rise of Agent and Coding workloads is challenging this classic assumption. According to statistics published by DeepSeek^Ad, the KV Cache hit rate for input tokens is around 56%, but in Agent + Coding scenarios it can reach 90% or even 95%. When hit rates are that high, Prefill only needs to compute the non-cached suffix, causing arithmetic intensity to plummet — Prefill is no longer necessarily compute-bound.

The seminar used a simplified model to quantify the performance ceiling for Decode: for a 70B BF16 model, generating one token requires loading roughly 140 GB of weights. Even with H100's full 3.35 TB/s HBM bandwidth, the per-token latency is at least 42 ms, yielding a theoretical TPS ceiling of about 24 tokens/second. This figure is far from satisfying, and it is the starting point for all the optimizations discussed below.

KV Cache: The Recurrent State of Transformers

KV Cache is the central concept for understanding inference systems. The seminar raised three key questions: Why is caching valid? Why cache K and V specifically? How do you recover them if they're discarded?

The validity of caching stems from the Causal Mask — new tokens depend on all preceding tokens, but that dependency never changes as new tokens are added, so historical K and V values can be safely cached. Only K and V are cached (not Q) because Q is only used at the current step. Once KV is discarded, recovery requires either recomputation or offloading from a multi-tier storage hierarchy, each with a distinct cost trade-off.

The size of the KV Cache is model-dependent. For Llama 3.3 70B with GQA, it's roughly 320 KB per token; the MLA architecture, which uses a 512-dimensional latent plus 64-dimensional RoPE design, can reduce the KV Cache by roughly an order of magnitude, delivering an enormous cost advantage. This explains why attention architecture evolution — from MHA to GQA to MLA to sparse attention DSA — is fundamentally about optimizing KV Cache storage and memory access overhead.

The reason KV Cache stores only Key and Value rather than Query is deeply tied to the computational structure of the Transformer attention mechanism. In each autoregressive decoding step, the attention formula is Attention(Q, K, V) = softmax(QK^T / √d) · V, where the current step's Query is only relevant to the current position and can be discarded afterward; the historical K and V, however, are re-accessed at every subsequent step (to compute attention scores between the current Q and all historical positions). Caching K and V therefore avoids redundant linear projection over historical tokens, at the cost of HBM occupancy. GQA (Grouped Query Attention) improves upon the original MHA (Multi-Head Attention) by having multiple Query heads share a single set of K/V heads, directly reducing KV Cache storage proportionally. MQA (Multi-Query Attention) is the extreme case of GQA, where all Query heads share a single K/V pair. MLA (Multi-head Latent Attention, proposed by DeepSeek) uses low-rank compression to compress KV into a low-dimensional latent vector before caching, dramatically reducing KV Cache storage overhead — currently one of the most aggressive KV compression approaches in the industry.

Core Serving Metrics and Workload Characteristics

Evaluating a serving system requires multiple dimensions. Throughput determines the provider's per-token cost, while latency determines individual user experience. The two are typically in tension — latency and throughput often trace an inverse curve. Academia tends to obsess over minimizing latency, while industry is more focused on "good-enough latency, maximize throughput."

Key metrics include TTFT (Time To First Token, affecting queue and first-token experience), TPOT (Time Per Output Token, measuring decode performance), ITL (inter-token latency distribution), and end-to-end latency. Production environments also need to track P50, P90, and P99 percentiles rather than just averages.

Workloads fall into two categories: online workloads with latency constraints, where only latency-compliant throughput counts as "good throughput"; and offline workloads, where latency constraints can be relaxed to push decode toward compute-bound territory. The seminar repeatedly emphasized that serving is a highly case-by-case topic — different scenarios (chat, document summarization, Agent Coding) fundamentally change the system's characteristics and the optimal optimization strategy.

PagedAttention and Cache Management

PagedAttention, introduced by vLLM, is "where the dream began" for inference systems. It borrows the virtual memory paging concept from operating systems to manage KV Cache, reducing the massive internal fragmentation caused by pre-allocating memory based on max sequence length down to at most one block's worth of waste — dramatically improving usable batch size. This requires maintaining a block table that maps logical blocks to physical blocks, and the attention kernel must explicitly support this paged layout.

Page size is a classic trade-off: larger pages benefit memory coalescing and TMA transfers (H100's TMA needs ~2 KB to saturate bandwidth) but increase internal fragmentation; smaller pages reduce fragmentation but hurt transfer efficiency. Additionally, Prefix Caching enables KV reuse across requests — vLLM uses block hashing while SGLang uses a radix tree; both designs have their merits, but real-world performance must be benchmarked on specific workloads.

HBM is an extremely expensive resource, and KV Cache flows through a multi-tier hierarchy: HBM → CPU DRAM → local NVMe → remote storage pools. Whether offloading is worthwhile depends on comparing transfer time against recomputation time and whether SLOs are met. Two intuitive conclusions hold: smaller KV Caches make transfer more attractive; transfers should be overlapped with HBM computation.

Before PagedAttention was introduced, mainstream inference frameworks (such as FasterTransformer) typically pre-allocated contiguous KV Cache memory for each request based on max_seq_len. This caused severe internal fragmentation: if a request only used 30% of its allocated length, the remaining 70% of memory sat idle until the request ended. External fragmentation between requests made it difficult for the scheduler to fit more concurrent requests. vLLM borrowed Linux's virtual memory paging mechanism, splitting KV Cache into fixed-size physical blocks (typically 16 or 32 tokens per block) and dynamically allocating blocks on demand, maintaining a block table to map logical sequences to physical blocks. This design confines fragmentation to within a single block, raising memory utilization from the typical 60–80% to near 100%, while naturally supporting physical block sharing across requests — laying the groundwork for Prefix Caching and Copy-on-Write (used in Beam Search).

PD Disaggregation and Parallelism Strategies

Because Prefill and Decode are so different in nature, mixing them together causes mutual interference — a long Prefill can starve ongoing Decode operations, leading to SLO violations. There are two mitigation approaches: Chunked Prefill (splitting prefill into chunks scheduled within a token budget) and PD Disaggregation (deploying the two phases on separate resource pools).

The cost of PD Disaggregation is the need to explicitly transfer KV Cache from Prefill instances to Decode instances. The seminar ran the numbers: a CX7 NIC at full 400 Gbps IB bandwidth delivers roughly 50 GB/s, so transferring 40 GB of KV Cache takes 0.8 seconds — unacceptable under certain TTFT constraints. Industry solutions like Mooncake and NIXL implement P2P transfers using layer-by-layer pipelining and RDMA one-sided operations to hide communication latency.

The choice of parallelism strategy is jointly determined by model architecture, KV Cache size, workload type, SLO requirements, and hardware interconnects. A common combination is: DP for Attention and EP for MoE in a large-EP configuration (DeepSeek V3's decode unit uses 144-card EP, amortizing weight load overhead across large batches and freeing up HBM to increase batch size); long-context scenarios require Context Parallelism (e.g., Ring Attention); TP and PP should stay within the NVLink domain.

RDMA (Remote Direct Memory Access) is the key underlying technology for KV Cache transfer in PD Disaggregation. Traditional TCP/IP transfers go through the OS kernel network stack, incurring high latency and CPU overhead. RDMA allows the NIC to bypass the CPU and OS and move data directly between the memory of two machines, reducing latency to microseconds with virtually no CPU involvement. InfiniBand (IB) is the most common RDMA network in data centers, with CX7 NICs supporting 400 Gbps bandwidth. RDMA one-sided operations (RDMA Read/Write) require no response from the remote CPU — the initiator can directly read or write the remote memory, further reducing coordination overhead. Mooncake (open-sourced by Moonshot AI) and NIXL (open-sourced by NVIDIA) build on this foundation to provide high-level P2P KV Cache transfer interfaces supporting layer-by-layer pipelined transfer: as soon as the Prefill instance finishes computing KV for one layer, it immediately sends it, overlapping with computation of subsequent layers — minimizing the impact of transfer latency on TTFT.

Speculative Decoding: Breaking the One-Token-Per-Step Assumption

Decode's memory-bound nature creates a clever optimization opportunity: since the memory access cost of loading weights once is fixed regardless, can we generate multiple tokens per step? Speculative Decoding does exactly this — a draft model proposes K tokens, and the large model verifies them all in a single forward pass. Since the bottleneck is memory-bound, the additional compute is nearly free, and there is a rigorous mathematical proof that the output distribution is lossless.

Draft models have evolved from standalone small models, to Medusa, to MTP and EAGLE. MTP's advantage is that the MTP heads are trained alongside the main model, so open-source models can benefit immediately. EAGLE supports tree-based drafting. This year, DeepFlash pioneered a new direction using Block Diffusion for parallel drafting, generating an entire block of candidates in a single forward pass.

Speculative Decoding trades off against batch size: at small batch sizes with abundant compute, draft benefits are significant; at large batch sizes approaching compute-bound territory, the draft's computational overhead may not be worth it. DeepSeek's DSpark design dynamically adjusts speculative parameters to achieve Pareto-optimal decisions.

The core mathematical guarantee of Speculative Decoding comes from the Rejection Sampling mechanism. During the verification phase, the large model runs a parallel forward pass over the K tokens proposed by the draft model, computing the ratio of the large model's distribution to the draft model's distribution at each position, and stochastically accepting or rejecting each token. It can be rigorously proven that after this correction process, the final output distribution is exactly equivalent to independently sampling from the large model — i.e., Speculative Decoding is lossless. The acceptance rate is the key factor affecting actual speedup: the higher the acceptance rate, the more valid tokens are produced per Draft-Verify cycle, and the greater the speedup. Methods like EAGLE significantly improve acceptance rates by having the draft model directly predict the large model's feature representations rather than token probabilities. Tree-based drafting maintains multiple candidate token paths simultaneously, submitting them as a tree structure for parallel verification by the large model, further increasing the expected number of tokens accepted per verification pass.

Insights from the Hardware Perspective

The seminar closed with a thought-provoking observation: all of the optimizations above implicitly assume the NVIDIA GPU ecosystem. When the binding constraints change, the conclusions may be entirely different. OpenAI chose to use GQA attention and design dedicated AI hardware from scratch, bypassing the need for PD Disaggregation while still achieving extremely high FLOPs per watt and bandwidth utilization.

This reminds infrastructure practitioners to take an evolving view of these problems — PD Disaggregation is an elegant solution under specific constraints, but hardware-software co-design for non-disaggregated scenarios is equally viable. The overall trajectory of inference system evolution is clear: scheduling granularity is becoming finer, resource management is becoming more precise, and model-system co-design is becoming deeper. For those working on Serving Infrastructure, the ultimate goal remains constant: given a trained model, achieve the lowest cost and highest throughput while satisfying SLO requirements.