From Enter to First Token: Decrypting the Full LLM Inference Pipeline (Autoregression, KV Cache & Decoding Strategies)

When you type a question into the ChatGPT input box and press Enter, what exactly happens inside the model before the first character appears on screen? This seemingly instantaneous process conceals a series of sophisticated engineering and algorithmic mechanisms—autoregressive generation, KV caching, sampling strategies, and more. Based on the finale of an LLM tutorial series for software engineers, Using the Trained Model, this article systematically walks through the core principles of the LLM inference stage.

Generating One Word at a Time: The Essence of Autoregression

Many people assume that large language models spit out an entire answer "all at once," but in reality, LLMs generate token by token. A token can be roughly understood as a text fragment like a word, subword, or punctuation mark.

The model works like this: it receives the complete prompt, and after a single forward pass, outputs a probability distribution over the vocabulary indicating the likelihood of each token being the "next word." One token is selected from this distribution and appended to the end of the input sequence. This longer sequence is then fed back into the model to predict the token after that. This loop continues until an end-of-sequence marker is generated or the length limit is reached.

This is autoregressive generation—where the output at each step depends on all previously generated content. This also explains why you see text "popping out" one piece at a time, rather than an entire passage appearing instantly.

Background: The Historical Roots of Autoregression The concept of autoregressive models far predates modern LLMs, with roots tracing back to the era of statistical language models. The term "autoregressive" comes from time series analysis, meaning the current value is linearly predicted from several past values. In the language modeling domain, the GPT series pushed this idea to its extreme: the model only needs to predict a single simple target—the "next word"—and through repeated training on massive amounts of data, capabilities for understanding, reasoning, and generation naturally emerge. Unlike bidirectional encoders represented by BERT (which look at text on both sides simultaneously), GPT-type models can only see the history on the left. This unidirectional constraint is precisely the prerequisite for autoregressive generation—it is both an architectural limitation and an engineering advantage: during generation, there's no need to wait for "future" information, allowing each token to be output immediately.

Why LLM Inference Is Slow

The serial nature of autoregression is an inherent bottleneck for LLM inference speed. The training phase can process an entire batch of data in parallel, but during generation, each token must wait until the previous token is finalized before computation can begin—true parallelization is impossible.

Worse still, in a naive implementation, for every new token generated, the model must redo the entire forward computation over an increasingly long full sequence. The longer the sequence, the greater the per-step computation load, causing noticeable slowdowns later in generation. This repeated computation is a massive waste—and the KV cache was born to solve exactly this problem.

KV Cache: Ending Redundant Labor in Inference

The KV Cache is a key technique for accelerating modern LLM inference. To understand it, we need to return to the Transformer's attention mechanism.

In attention computation, each token generates three vectors: Query, Key, and Value. The core of attention is using the current token's Query to match against all historical tokens' Keys, then weighting and aggregating the corresponding Values accordingly.

Deep Dive: The Mathematical Foundation of KV Cache The Transformer's multi-head self-attention mechanism is the theoretical foundation that makes the KV cache possible. In each attention head, every position in the input sequence is projected into three sets of vectors—Query, Key, and Value—through three independent linear transformation matrices (W_Q, W_K, W_V). Attention scores are computed via the dot product of Query with all Keys, normalized through softmax, and then used to compute a weighted sum of Values. The key point is this: for all tokens before position i, once their Key and Value vectors are computed, they do not change in subsequent steps (because historical tokens in an autoregressive model are never modified). The KV cache exploits precisely this invariance, persisting the K and V matrices of every attention head in every layer in GPU memory, avoiding redundant matrix multiplications, and reducing the time complexity of inference from O(n²) to an incremental computation pattern approaching O(n).

The key insight is: once a token has been generated, its Key and Value vectors are fixed and unchanging. Given this, why recompute them at every step? The idea behind the KV cache is to store these already-computed Keys and Values, so that when generating the next token, you only need to compute the new token's own Q, K, and V, then retrieve the historical K and V from the cache.

This reduces the per-step computation from "processing the entire sequence" to "processing just one new token," dramatically boosting generation speed. The cost is memory usage—the KV cache grows linearly with the sequence length, which is a major reason why GPU memory becomes tight in long-context scenarios.

Engineering Reality: The Scale Challenge and Cutting-Edge Optimizations of KV Cache Take LLaMA-3 70B as an example: each token's KV cache across all layers occupies several MB of GPU memory, and processing a 32K context can consume tens of GB of memory just for the KV cache. To address this, the industry has developed several optimization techniques: PagedAttention (proposed by vLLM) borrows the paging concept from operating system virtual memory, splitting the KV cache into fixed-size blocks stored non-contiguously, significantly improving memory utilization under concurrent multi-request workloads; Grouped-Query Attention (GQA) compresses cache size by having multiple Queries share the same set of K and V (adopted by mainstream models like LLaMA-3 and Mistral); and quantized KV cache (e.g., INT8/FP8 precision storage) directly halves storage overhead. Together, these techniques form the core competitiveness of modern LLM inference engines and are a major source of performance differences between inference frameworks like vLLM and TensorRT-LLM.

Decoding Strategies: How the Model "Picks Words"

At each step, the model outputs a probability distribution—but which specific token gets chosen? This is the job of the decoding strategy, which directly determines the style of the response: whether rigorous and deterministic, or wildly imaginative.

Temperature: The Dial for Randomness

Temperature is the most common control parameter. It scales the model's output logits before they pass through softmax normalization.

• Low temperature (close to 0): Amplifies the advantage of high-probability tokens, sharpening the distribution. The model tends to pick the most likely word every time, producing more deterministic, conservative output—suitable for factual Q&A and code generation.
• High temperature (greater than 1): Flattens the probability distribution, giving low-probability tokens a chance to be selected, producing more diverse and creative output—but also more prone to "saying nothing meaningful."

When temperature is 0, this essentially degenerates into greedy decoding—taking only the highest-probability token at each step.

Background: The Physics Origin of the Temperature Parameter The mathematical form of the temperature parameter is directly borrowed from the Boltzmann distribution in statistical physics. The standard softmax formula is P(i) = exp(z_i) / Σexp(z_j); introducing temperature T changes it to P(i) = exp(z_i/T) / Σexp(z_j/T). As T approaches 0, the probability corresponding to the maximum logit approaches 1, forming a one-hot distribution; when T=1, it's the standard softmax; and as T approaches infinity, all token probabilities converge to a uniform distribution. In physical systems, temperature likewise controls the probability distribution of particles across different energy states—at low temperatures the system tends toward the lowest energy state (equivalent to greedy selection), while at high temperatures all states become roughly equally probable (equivalent to random sampling). This physical intuition helps explain why a model at temperature=0.1 produces almost always consistent output, whereas at temperature=1.5 the model becomes "wildly imaginative." In engineering practice, the OpenAI API's temperature parameter ranges from 0 to 2; most creative writing tasks recommend 0.7–1.0, while code and mathematical reasoning are best served by 0–0.2.

Top-k and Top-p: Truncating the Sampling Range

Beyond temperature, there are two other mainstream strategies for controlling the candidate range:

Top-k sampling: Samples only from the k highest-probability tokens, excluding all others. For example, with k=50, selection is restricted to the top 50 candidates regardless of the situation. It's simple and direct, but the limitation of a fixed k is obvious—sometimes the top few tokens already account for the vast majority of the probability mass, so extra candidates only introduce noise; other times the distribution is very flat, and k may restrict a legitimate range of choices.

Top-p sampling (nucleus sampling): Dynamically selects the smallest set of tokens whose cumulative probability reaches a threshold p. For example, with p=0.9, sampling is done from the group of tokens that, accumulated from highest to lowest probability, just exceeds 90%. It adaptively adjusts the number of candidates based on how "steep" the current distribution is, making it more flexible than Top-k, which is why it's more widely used in practice.

Academic Background: The Proposal of Nucleus Sampling and Research on "Text Degeneration" Top-p nucleus sampling was formally proposed by Holtzman et al. in the 2020 paper The Curious Case of Neural Text Degeneration, which systematically studied the phenomenon of "degeneration" in language model generated text—including meaningless repetition (repetition loops), self-contradiction, and semantic drift. The researchers found that at each generation step, the probability mass is often highly concentrated on a small number of tokens, but exactly how many tokens it concentrates on varies dynamically depending on the context. Top-k's fixed truncation cannot adapt to this variation: when the distribution is sharp, k=50 may include 99% low-quality long-tail candidates; when the distribution is flat, k=50 may wrongly exclude a large number of reasonable options. Nucleus sampling dynamically adjusts the candidate set size to match the distribution's "effective support set," achieving a better balance between generation quality and diversity. This paper is one of the most widely cited engineering studies in the NLP field, directly driving the industry convention of modern LLM APIs exposing top_p as a core parameter to users.

In actual deployment, these strategies are often used in combination: first adjust the distribution with temperature, then truncate the range with Top-p or Top-k, and finally sample. Understanding these parameters helps developers precisely control LLM behavior—lower the temperature for accuracy, loosen it appropriately for creativity.

Reviewing the Complete Inference Pipeline

Stringing the entire LLM inference process together:

1. Input processing: The prompt is split into tokens and converted into embedding vectors.
2. Forward pass: Passing through multiple Transformer layers to output the probability distribution for the next token.
3. Decoding and sampling: Temperature, Top-k, and Top-p together determine which token is chosen.
4. Autoregressive loop: The new token is added to the sequence, and with the help of the KV cache, the above process is efficiently repeated until generation ends.

From pressing Enter to seeing the first character, the model completes one full forward computation and sampling; every subsequent character is a rapid iteration of this loop. Notably, there's an important stage division in this process: the phase of processing the input prompt is called prefill, during which the KV values of all input tokens can be computed in parallel; the phase of generating new tokens step by step is called decode, constrained by the serial nature of autoregression. Modern inference engines (such as vLLM and SGLang) apply different optimization strategies to these two phases—the prefill phase is more compute-bound, while the decode phase is more memory-bandwidth-bound.

Final Thoughts

This article is the fourth part (the finale) of this tutorial series. The previous three parts covered, respectively: Part 1 text processing (tokenization, embedding, and forward pass); Part 2 model learning (loss functions, backpropagation, and optimizers); and Part 3 from toy models to GPT (scaling, parallelism, fine-tuning, RLHF, and DPO).

For software engineers, understanding LLM inference mechanisms isn't just intellectual satisfaction—it directly relates to engineering practice: why long contexts are expensive (KV cache memory usage), how to tune parameters to control output quality (decoding strategy selection), and how to optimize inference services (batching, cache reuse). When you truly understand "what happens after pressing Enter," you'll gain a greater sense of control when using and deploying large language models.