How an LLM Generates a Word: The Complete Pipeline from Token to KV Cache

Every time a large model emits a character, it runs through an entire precise pipeline behind the scenes. An AI agent broke down this process in a blog post: first the text is split into tokens, then it passes through the attention and per-token feedforward computations of multiple transformer layers, and finally the output head scores and samples over the vocabulary. The core loop of this whole process can be summed up in one sentence—predict, append, predict again, append again.

Tokens Are Not Words—What the Model Actually "Sees"

What's fed into the model is never words, but tokens. Text is split into tokens that are usually word fragments—for example, "The quick brown fox" might be split into "The" "quick" "brown" "fox", with each token mapping to an ID in a fixed vocabulary. All of the model's subsequent "understanding" starts from these IDs. This also explains why spelling errors and the way rare words are split can directly affect performance—change the split result, and the input sequence changes.

Vocabulary and Tokenizer

The so-called "fixed vocabulary" is a set of symbols determined before training, typically ranging from tens of thousands to over a hundred thousand (e.g., the Llama series has about 128K, the GPT series around 100K). The tokenizer maps raw text to token IDs in the vocabulary, using common algorithms like BPE (Byte Pair Encoding), WordPiece, and SentencePiece. BPE starts from characters and repeatedly merges the most frequent adjacent fragments in the corpus, ultimately achieving a balance where common words are kept whole and rare words are split—common words often occupy one token, while rare words, spelling variants, or cross-lingual text are split into multiple subword fragments. This directly determines the "billing unit" and "understanding granularity" of the same string: a single spelling error might turn what was 1 token into 3, both lengthening the sequence and changing the input the model sees. Thus vocabulary size and tokenization strategy affect the model's compression efficiency, multilingual performance, and the actual usable length of the context window.

What Happens Inside Transformer Layers

Internally, the model is a stack of transformer layers, each doing two things: attention mixes information between tokens, while the feedforward network transforms each token independently.

Attention is the key to transformer's success. Each token generates three vectors—query (what I'm looking for), key (what I can offer), and value (the information I actually carry). When a token's query matches another token's key, attention is paid to the latter; after normalizing the scores, the values are mixed by weight, and the current token's representation is updated.

In a decoder-only architecture, causal masking ensures each token can only see itself and earlier tokens, never "peeking" ahead. This is precisely why "bank" means different things in "river bank" and "bank account": the same token ID, with different context, gets different attention allocations, and the semantics diverge.

Multi-Head Attention and Positional Encoding

Real models don't just do attention once; they split query/key/value into multiple "heads" computed in parallel, each learning different attention patterns in different subspaces—some heads focus on nearby words, others track syntactic dependencies or coreference relations—and finally concatenate the head outputs and apply a linear transformation. This is "multi-head attention." Moreover, attention itself is insensitive to order, so positional information must be injected separately. A common approach is Rotary Position Embedding (RoPE), which encodes relative positions into attention scores by rotating query and key vectors, enabling the model to distinguish "A before B" from "B before A." The design of positional encoding directly affects the model's ability to extrapolate to long contexts, and is also the part that long-context models need to specifically adjust.

Prefill and Decode: Two Phases, Two Kinds of Overhead

Actual inference is divided into two phases with completely different natures.

Prefill phase: The entire prompt is processed at once, with the model using large-scale matrix operations in parallel—this is the compute-intensive part. Meanwhile, the model stores the key and value vectors for each layer and each token into the KV cache.

Decode phase: Only one token is generated at a time. At each step, only the new token's key and value are computed; everything else is read from the cache rather than recomputed. Without the cache, generating the 1000th token would require recomputing the key and value of all previous 999 tokens—at every step, an unacceptable overhead. The cache is the lifeblood of generation speed, and also the source of cost.

Why KV Cache Is the Most Expensive Part of Long Context

The cache grows linearly with each token in the context, and its size is staggering. Calculated as 2 (key and value) × layers × KV heads × head dimension × sequence length × bytes per value, Llama 3.1 70B at 128K context and half precision has a KV cache of about 43GB per request—before generating the first word, it already consumes more than half of an 80GB GPU.

This is the structural reason long context is expensive: providers charge more for long context, while model developers keep inventing tricks to shrink the cache, such as Grouped Query Attention (GQA), quantization, and cache offloading. According to Modular's LLM inference handbook and Sebastian Raschka's KV Cache calculations, these optimization directions directly determine the inference cost curve.

Grouped Query Attention (GQA), Quantization, and Offloading

The core idea of GQA is to let multiple query heads share the same set of key/value heads, greatly reducing the "KV heads" multiplier in the KV cache. Taking Llama 3.1 70B as an example, it uses 8 KV heads instead of 64 attention heads, reducing the cache size to about one-eighth, with minimal loss in quality. Quantization reduces the "bytes per value": switching from half precision (2 bytes) to INT8 or even INT4 can shrink the cache by another 2 to 4 times, at the cost of precision and implementation complexity. Cache offloading moves inactive KV blocks to CPU memory or disk, trading bandwidth for VRAM. These three approaches differ in direction but share the same goal—flattening the curve that grows linearly with sequence length—and together they form the main line of current long-context inference cost optimization.

The Most Counterintuitive Point: The Model Doesn't Remember You at All

With every round of conversation, the entire history is resent as input. The model has no memory between requests; the so-called context window is just a piece of text fed in repeatedly. The "memory" you perceive is reconstruction, not storage. The reason a conversation seems coherent is simply that the complete history is carried along each time.

This also means that the longer the context, the heavier the burden of repeated computation—combined with the KV cache cost from the previous section, this constitutes the core contradiction in current large model service costs.

Summing Up the Loop in One Sentence

A large model is a next-word predictor: it uses attention to weigh what it reads, uses cache to trade for speed and also to trade for cost, and remembers nothing except what you feed it again. Seeing this loop clearly, it is both awe-inspiring and surprisingly simple.

The mechanism descriptions in this article cross-reference Modular's inference handbook, Vaswani et al.'s "Attention Is All You Need" (2017), Sebastian Raschka's KV Cache calculations, and Spheron's optimization guide.

One-line Summary

Breaking down the complete per-word generation pipeline of LLMs from tokenization, attention, Prefill/Decode, to KV Cache.

Paragraph Summary

The article takes "predict, append, predict again" as its core loop and thoroughly dissects the entire process of a large model generating a word: text is first split into tokens by a tokenizer and mapped to IDs; inside stacked transformer layers, attention mixes information between tokens via query/key/value matching, with causal masking ensuring only history is seen; inference is split into Prefill and Decode phases—the former processes the prompt in parallel and writes to the KV cache, the latter generates token by token relying on the cache to avoid recomputation. The article highlights that the KV cache grows linearly with context, with Llama 3.1 70B at 128K context and half precision being about 43GB—a structural reason long context is expensive—and introduces optimization directions like GQA, quantization, and offloading. Finally, it emphasizes that the model has no memory between requests, and conversational coherence comes from repeatedly feeding in the history.