A Deep Dive into the vLLM Inference Framework: Core Principles of Throughput Optimization and Interview Guide

Introduction: Why Is LLM Inference Optimization a High-Frequency Interview Question

In the process of engineering and deploying large language models, inference throughput optimization is an almost unavoidable core topic, and it's also a classic question repeatedly probed in algorithm engineer interviews. To truly answer this question well, we need to clarify three things: What is throughput? What is the essence of LLM inference? And how do we optimize the inference process?

Starting from the most fundamental concepts, this article breaks down layer by layer the core principles of vLLM—a mainstream inference framework—to help readers with zero background build a complete knowledge framework.

Extended Background: The Origins of vLLM vLLM was open-sourced by the Sky Computing Lab at UC Berkeley in 2023, and its paper "Efficient Memory Management for Large Language Model Serving with PagedAttention" was published at SOSP 2023. In its launch benchmarks, vLLM achieved up to a 24x throughput improvement compared to the native inference implementation of HuggingFace Transformers, and more than a 3.5x improvement over the already-optimized FasterTransformer at the time, quickly becoming the framework of choice for deploying LLM inference services in industry. Its core contribution, PagedAttention, has been widely adopted and implemented by mainstream inference frameworks such as TensorRT-LLM, LMDeploy, and SGLang.

It's worth adding that vLLM's open-sourcing coincided with a critical juncture in the large-scale deployment of LLM inference services—at that time, ChatGPT had ignited the market, and major cloud vendors and startups alike faced the engineering challenge of serving massive concurrent requests with limited GPU resources. By migrating the classic ideas of operating system memory management to GPU memory management, vLLM provided an elegant and efficient solution, rapidly gaining industry recognition.

Part One: Understanding Throughput — Why Use tokens/s Instead of QPS

Throughput refers to the number of requests a system can process and complete per unit of time. Some students interpret it as "QPS request handling capacity," and this direction is correct—throughput measures exactly the system's request processing rate.

But for large language model (LLM) inference, the unit of measurement for throughput needs to be more granular. We typically measure not in terms of "number of requests" but in terms of number of tokens—that is, "how many tokens can be processed per unit of time." The reason is simple: different requests vary enormously in input and output length, so using tokens as a unified unit of measurement more accurately reflects the system's true processing capacity.

This detail is crucial: "tokens/s" is the core metric for evaluating the performance of an LLM inference framework, not simply the number of requests.

Extended Background: The Engineering Significance of Token Measurement In actual engineering, token measurement is not only used to gauge throughput but also directly affects the billing model and resource planning of inference services. Mainstream LLM API services (such as OpenAI and Anthropic) all bill by token count, because it more fairly reflects the actual computational consumption. Furthermore, the tokens/s metric is subdivided into two dimensions: Time to First Token (TTFT) measures the response time a user waits for the first output, directly related to user experience; Generation Throughput measures the output speed of subsequent tokens, which determines overall inference smoothness. Proactively distinguishing these two sub-dimensions in an interview often leaves a strong impression on the interviewer.

In addition, differences in tokenizer efficiency across models also affect actual throughput perception—for the same natural language text, the tiktoken of GPT-4o and the SentencePiece of LLaMA may produce tokenization results that differ by 10%–20%, which needs to be normalized when comparing performance across models.

Part Two: The Essence of LLM Inference — Autoregressive Generation

The Token-by-Token Prediction Generation Process

To optimize inference, we must first understand what LLM inference is actually doing. In essence, a large model is continuously predicting the next token.

A token is the basic unit through which a large model processes text, produced by a tokenizer that splits raw text. Common tokenization algorithms include BPE (Byte Pair Encoding), WordPiece, and SentencePiece. Taking the BPE used by the GPT series as an example, the English word "running" might be split into two tokens, "run" and "ning," while Chinese is usually split by character or character groups. Generally speaking, 1000 English tokens correspond to roughly 750 English words. This granularity design strikes a balance between vocabulary size and expressive power—the vocabulary typically ranges between 32,000 and 100,000 tokens, which can cover most linguistic phenomena without letting the model's parameter scale spiral out of control. Understanding token granularity also helps engineers estimate memory requirements and throughput ceilings in actual deployment.

Let's illustrate the entire generation process with a classic example, assuming the initial input is "the dog":

• Step 1 (time step 1): Feed "the dog" into the model, which predicts the next word, say "sat";
• Step 2 (time step 2): Concatenate the output back into the input, so the new input becomes "the dog sat," and the model continues predicting, yielding "down";
• Step 3 (time step 3): The input further becomes "the dog sat down," and the model outputs the special identifier EOS (End of Sentence).

When the model outputs EOS, generation ends. The final output is "the dog sat down," and EOS itself is not printed.

Extended Background: The Mathematical Essence of Autoregressive Generation From a probability-theory perspective, autoregressive generation can be expressed as: the model performs a chain decomposition of the joint probability of the text sequence, i.e., P(x₁, x₂, ..., xₙ) = ∏P(xᵢ | x₁, ..., xᵢ₋₁). Each generation step samples (or takes the argmax of) the next token from the probability distribution over the vocabulary, conditioned on the historical context. This mathematical structure dictates that the generation process is inherently sequential—the output of step i must wait for step i-1 to complete, and cross-step parallelism cannot be achieved in the Decode phase. This stands in sharp contrast to the Prefill phase: when Prefill processes user input, the positions of all input tokens are known, so the attention matrix can be computed fully in parallel, making it a compute-intensive operation.

The Performance Bottleneck Brought by Autoregression

This mechanism of "predicting the next token based on everything preceding it at each step" is called autoregressive generation. It is closely tied to the Decoder architecture of the Transformer: at each generation step, the model needs to use the attention mechanism to compute the correlation weights between the current token and all historical tokens in the sequence, and this computation has a time complexity of O(n²), where n is the sequence length.

Extended Background: Why Is Attention Computation O(n²)? In standard Scaled Dot-Product Attention, let the sequence length be n and the hidden dimension be d. The Query matrix Q (shape n×d) and the Key matrix K (shape n×d) undergo matrix multiplication to produce an n×n attention score matrix, and this step has a computational cost of O(n²·d). As the sequence length n grows, the computational cost grows quadratically. For this reason, the industry has been exploring more efficient attention variants, such as FlashAttention (which dramatically reduces memory bandwidth consumption through IO-aware tiled computation) and linear attention (which reduces complexity to O(n)). But in current mainstream LLM deployments, standard Multi-Head Attention remains dominant, and the O(n²) bottleneck is still very real.

Extended Background: Differences in Hardware Characteristics Between the Prefill and Decode Phases Autoregressive inference is actually divided into two phases with completely different characteristics. The Prefill phase processes the user's input prompt, where the attention for all input tokens can be computed in parallel, making it a compute-bound operation. The GPU's CUDA Cores and Tensor Cores are fully utilized, and compute utilization can reach 60%–80%. The Decode phase generates output tokens one by one, where each step only needs to compute one new token but must read the KV Cache of all historical tokens, making it a memory-bandwidth-bound operation, with GPU compute severely idle. This fundamental difference has given rise to the recent "Prefill-Decode disaggregation" (PD disaggregation) architecture—scheduling the two types of workloads onto different types or configurations of GPU clusters, optimizing for compute and bandwidth respectively. This is currently a cutting-edge practice for large-scale inference clusters.

This means that as the generated sequence grows, the computation per step grows quadratically, and the amount of redundant computation expands at a linear or even higher rate. Every time a new token is generated, the entire previously generated sequence must be reprocessed. This is the root of the LLM inference performance bottleneck, and it is the core problem that inference optimization frameworks such as vLLM focus on tackling.

Part Three: How vLLM Optimizes Inference Throughput — Three Core Technologies

Having understood the essence of autoregressive inference, the direction for optimization becomes clear. vLLM achieves high-throughput inference primarily through the following three key technologies.

1. KV Cache: Eliminating Redundant Computation

The most direct optimization idea is: cache the intermediate results that have already been computed, avoiding computing from scratch at each step. This is the core idea of the KV Cache (Key-Value Cache).

The design of the KV Cache is based on the mathematical properties of the Transformer attention mechanism. In standard Multi-Head Attention, each token generates three vectors: Query (Q), Key (K), and Value (V). When computing attention, the Q of the current token needs to do a dot product with the K of all historical tokens, then use the resulting weights to perform a weighted sum over all historical V's. The key point is: the K and V vectors of historical tokens do not change when generating a new token, so they can be safely cached and reused. The KV Cache stores these already-computed K and V matrices in GPU memory, so that when generating each new token, only its own Q, K, and V need to be computed, rather than recomputing the entire historical sequence, reducing the per-step computational complexity from O(n²) to O(n).

Extended Background: Quantitative Analysis of KV Cache Memory Consumption The memory footprint of the KV Cache can be estimated precisely. For a model with L layers and a hidden dimension of d, the KV Cache size per token is approximately 2 × L × d × sizeof(dtype) bytes (the 2 represents the K and V matrices). Taking LLaMA-2-7B as an example (32 layers, d=4096, FP16 precision), each token occupies about 512KB. If the system needs to support 100 concurrent requests, each averaging 2048 tokens, then the total KV Cache is about 100GB, approaching the memory ceiling of an A100-80G. This estimate reveals a core contradiction: the model parameters themselves (7B × 2 bytes ≈ 14GB) account for only a small fraction of the memory, while the KV Cache is the main consumer of memory in high-concurrency scenarios. This is precisely the fundamental motivation for PagedAttention's efforts to solve the memory utilization problem.

However, the KV Cache also brings a new challenge—as the sequence length increases, the memory occupied by the KV Cache also grows linearly. For long-context scenarios (such as a 128K-token context window), the continuous growth and fragmentation of memory are exactly the problems that subsequent technologies need to further solve.

2. Continuous Batching: Maximizing GPU Utilization

Batching is a classic means of improving throughput, i.e., processing multiple requests simultaneously to fully leverage the GPU's parallel computing capability. But traditional static batching packs a group of requests into a fixed batch and waits until all requests are complete before processing the next batch, which has an obvious efficiency shortcoming: different requests vary in generation length, potentially by tens of times—some requests only need to output 20 tokens, while others need to output 2000. After short requests complete, they must wait for the longest request in the same batch to finish, causing a large amount of GPU resources to spin idle and go to waste.

Extended Background: The Relationship Between GPU Utilization and Inference Efficiency The root of static batching's inefficiency lies in the "barrel effect": batch throughput is determined by the slowest request. If a batch contains one long request that needs to generate 2000 tokens, even if the other 31 requests are already done within 100 steps, the entire batch must wait 2000 steps to be released, causing the GPU to process only 1/32 of the workload during the final 1900 steps. The NVIDIA A100's theoretical FP16 peak compute is about 312 TFLOPS, but in naive LLM inference scenarios, GPU utilization is often below 30%. Continuous batching completely breaks this limitation through "iteration-level scheduling": after each forward pass completes, the scheduler immediately checks for completed requests and fills in new ones, keeping batch fill levels near the maximum at all times. Measured data shows that in high-concurrency scenarios, continuous batching can raise GPU utilization from 30%–40% to 70%–85% compared to static batching, making it one of the most direct and effective scheduling strategies for improving throughput.

The continuous batching mechanism adopted by vLLM (also called iteration-level batching) refines the scheduling granularity down to each generation step: after each iteration completes, the scheduler checks which requests have finished (output EOS), immediately removes them from the batch, and fills in new requests from the waiting queue. This "remove-and-refill" mechanism keeps the GPU in a high-utilization state at all times, thereby significantly improving overall throughput.

3. PagedAttention: A Revolution in Memory Management

vLLM's most representative technical innovation is PagedAttention. It borrows the design philosophy of the operating system's "virtual memory paging"—in an operating system, physical memory is divided into fixed-size "pages," and a process's logical address space is mapped to physical pages through a page table, allowing memory to be stored non-contiguously, thereby greatly reducing external fragmentation.

PagedAttention brings the same idea to KV Cache management: it divides each request's KV Cache into equal-sized "blocks," each storing the K and V vectors of a fixed number of tokens, and blocks do not need to be stored contiguously in memory. The scheduler maintains a Block Table that records the mapping from each request's logical blocks to physical blocks. The KV Caches of different requests can flexibly share and reuse physical memory blocks, avoiding the massive memory waste caused by pre-allocating for the maximum sequence length in traditional approaches.

Extended Background: The Fragmentation Problem of Traditional KV Cache Management Before PagedAttention appeared, mainstream inference frameworks typically pre-allocated a contiguous memory block capable of holding the maximum output length at the start of a request. Since the actual output length cannot be known in advance, the system often reserved memory according to the worst case (maximum sequence length), causing a large amount of "internal fragmentation"—reserved space that was never used. Meanwhile, requests of different sizes were assigned contiguous memory blocks of different sizes, and as requests were continuously created and destroyed, a large amount of "external fragmentation" that could not be utilized by new requests appeared in memory. Measurements in the vLLM paper show that these two types of fragmentation caused traditional approaches to have an actual KV Cache utilization of only 20%–40%, with the rest of the memory wasted.

Extended Background: The Engineering Value of Copy-on-Write and Prefix Sharing PagedAttention's physical block sharing mechanism also gives rise to two important engineering features. The first is prefix caching: in RAG systems or dialogue scenarios with a fixed System Prompt, the KV Cache physical blocks corresponding to a shared prefix among multiple requests only need to be computed once, and subsequent requests can reuse them directly, reducing Prefill computation by 30%–70% in scenarios with highly repetitive prefixes. The second is the copy-on-write mechanism: when Beam Search or parallel sampling generates multiple candidate sequences, all candidates share the prefix physical blocks and only allocate new blocks after the sequences diverge, reducing memory consumption from O(beam_width × sequence_length) to near O(sequence_length). These two features make PagedAttention's actual benefit in complex inference scenarios far exceed simple fragmentation elimination, and they hold significant engineering value in practical scenarios such as RAG (Retrieval-Augmented Generation) and Beam Search.

Measured data shows that PagedAttention raises KV Cache memory utilization from below 40% to nearly 100%, supporting several times the number of concurrent requests of traditional approaches on the same hardware, ultimately bringing a significant leap in overall throughput.

4. Intelligent Scheduling: The Command Center That Integrates Everything

The intelligent scheduling module in the vLLM architecture is responsible for coordinating request admission, the allocation and reclamation of the KV Cache, and the dynamic organization of batches. It is precisely this scheduling mechanism that organically integrates the KV Cache, continuous batching, and PagedAttention, together forming a complete solution for high-throughput inference.

Extended Background: The Multi-Objective Trade-off Challenge of the Scheduler vLLM's scheduler needs to make real-time trade-offs among multiple conflicting objectives. There are three core tensions: first, the throughput vs. latency trade-off—larger batches yield higher throughput, but the queuing wait time per request also grows, affecting P99 latency; second, the memory utilization vs. OOM risk trade-off—aggressively admitting more requests improves utilization, but once memory is exhausted, costly preemption is triggered; third, the fairness vs. efficiency trade-off—prioritizing long requests avoids swapping out the KV Cache, but may cause short requests to starve for a long time. When memory is insufficient, the scheduler executes a preemption strategy: swapping out the KV Cache blocks of low-priority requests to CPU memory (Swap) or directly discarding and recomputing them (Recompute), each strategy carrying its own latency cost. The optimal solution to these trade-offs remains an active research direction in both academia and industry to this day; systems such as Sarathi-Serve and Orca have proposed different scheduling improvement schemes, and this is also an excellent entry point for demonstrating the depth of systems thinking in interviews.

Summary: Answer the Inference Optimization Interview Question in Three Steps

Returning to the initial interview scenario, as long as you clarify the following three levels, the answer naturally emerges:

1. What is throughput: The number of tokens processed per unit of time (tokens/s), the core metric for measuring the processing capacity of an LLM inference system; it can be further subdivided into two dimensions: Time to First Token (TTFT) and generation rate;
2. What is the essence of inference: Autoregressive token-by-token generation, which is essentially a chain decomposition of the sequence's joint probability, divided into the Prefill phase (parallel processing of input, compute-intensive) and the Decode phase (step-by-step generation, bandwidth-intensive), with the core bottleneck being the O(n²) redundant computation overhead brought by sequence growth;
3. How to optimize: Use the KV Cache to reduce the per-step computational complexity from O(n²) to O(n) to eliminate redundant computation; use continuous batching and intelligent scheduling to maximize GPU utilization (which can be raised from 30% to over 80%); use PagedAttention to solve memory fragmentation, raise KV Cache utilization to near 100%, and support advanced features such as prefix sharing and copy-on-write.

These three steps progress layer by layer, forming a complete logical chain for understanding modern LLM inference frameworks. Once you master this line of thinking, you will not only handle interviews with ease but also truly understand why vLLM has become the mainstream choice for LLM inference deployment.

Key Takeaways

Technology	Problem Solved	Core Mechanism	Effect
KV Cache	Autoregressive redundant computation	Cache historical K/V vectors	Per-step complexity O(n²)→O(n)
Continuous Batching	GPU idle waste	Iteration-level dynamic refilling of new requests	GPU utilization raised from ~30% to ~80%
PagedAttention	Memory fragmentation and waste	Non-contiguous blocking + block table mapping	KV Cache utilization raised from ~40% to near 100%
Intelligent Scheduling	Resource coordination and preemption	Dynamic priority + memory swap-out/recompute	Integrates the three technologies to maximize overall throughput