Deep Dive into LLM Mechanisms: Tokens, Attention, and KV Cache Dynamics

The inner workings of large language models (LLMs) are often treated as a black box, but breaking down their core mechanisms—tokenization, attention computation, and KV caching—can help developers understand model behavior and optimize inference performance. This article, based on an in-depth discussion in the Reddit community, outlines the underlying logic and engineering trade-offs of these key components.

Tokens: The Atomic Units of Text Understanding

LLMs do not process characters or words directly; instead, they split text into sequences of tokens. A token can be a word, a subword, or even a single character, depending on the tokenizer's algorithm. Common approaches like BPE (Byte Pair Encoding) balance vocabulary size and sequence length by statistically identifying frequent subwords.

Tokenization directly affects model capabilities: a vocabulary that is too small leads to longer sequences and higher computational costs; a vocabulary that is too large increases embedding layer parameters and sparsity. Differences in tokenization strategies across models also explain why the same text may yield different token counts in different models—which in turn affects the actual usable capacity of the context window and inference costs.

BPE (Byte Pair Encoding) is a data-driven tokenization algorithm originally used for data compression. Its training process starts with a character-level vocabulary and repeatedly counts the frequency of adjacent symbol pairs in the corpus, merging the most frequent pairs into new symbols until the vocabulary reaches a preset size. When used for LLMs, a common variant is byte-level BPE (as in the GPT series), which operates on bytes rather than Unicode characters, thereby losslessly representing any text and avoiding out-of-vocabulary issues. BPE's merge rules determine subword granularity: high-frequency words tend to become complete tokens, while rare words are split into multiple subwords. Thus, vocabulary size and corpus distribution jointly shape tokenization results, which also explains why multilingual or code text shows significantly different token inflation rates across models.

Attention Mechanism: The Core of Contextual Association

Self-attention allows each token to dynamically aggregate information based on its relevance to other tokens. Its computation can be summarized as: mapping the input into three sets of vectors—queries (Q), keys (K), and values (V)—computing attention weights via the dot product of Q and K, and then taking a weighted sum of V.

Standard attention complexity grows quadratically with sequence length (O(n²)), which is the main bottleneck in long-context scenarios. To address this, the community has produced various optimizations: sparse attention, sliding window attention, linear attention approximations, and more. The Reddit discussion emphasizes that understanding the distribution of attention weights helps diagnose what the model is "focusing on"—for example, some heads specialize in local syntax while others capture long-distance dependencies.

Multi-Head Attention is a parallelized extension of standard self-attention: Q, K, and V are each mapped through multiple independent linear projections into low-dimensional subspaces, attention is computed independently within each subspace, and the outputs of all heads are concatenated and fused through a linear layer. Different heads learn different association patterns due to different parameters—for example, some heads focus on local syntax of adjacent tokens, while others track long-distance coreference between pronouns and their antecedents. This is the mechanistic basis for the article's claim that "understanding the distribution of attention weights helps diagnose what the model is focusing on." Increasing the number of heads enhances representational capacity but also adds computation and caching overhead; in practice, a balance between efficiency and performance is often achieved through head pruning or Grouped Query Attention (GQA).

KV Cache: The Key to Inference Acceleration

During autoregressive generation, each new token requires computing attention with the K and V vectors of all previous tokens. Recomputing this every time would be enormously expensive. The KV cache stores the K and V vectors of historical tokens to avoid redundant computation, thereby significantly reducing generation latency.

But the KV cache is not free: its memory footprint grows linearly with sequence length, and it can become a VRAM bottleneck in long-context and large-batch inference. Approaches mentioned in the discussion include: quantized caching (storing at lower precision), paged management (such as the PagedAttention approach), and selectively discarding some historical cache. These trade-offs directly affect throughput and generation quality.

PagedAttention is a KV cache management scheme inspired by virtual memory paging in operating systems (proposed by vLLM). Traditional implementations pre-allocate contiguous VRAM blocks for each sequence; because the actual sequence length is unknown and varies, this often causes internal and external fragmentation and low VRAM utilization. PagedAttention divides the KV cache into fixed-size blocks, mapping logically contiguous KV entries to non-contiguous physical blocks via a block table, allocating on demand—thereby greatly reducing fragmentation and improving batch concurrency. The block table also supports copy-on-write sharing, allowing caches for identical prefixes to be reused in beam search or parallel sampling, further reducing VRAM usage and redundant computation.

Engineering Insights and Community Consensus

Drawing on the Reddit discussion, several points of consensus deserve attention: tokenization strategy must match the task; attention optimization must balance accuracy and efficiency; and KV cache management is a core engineering challenge for long-context inference. According to the analysis in the thread, understanding the interplay among these three is more effective for improving real-world deployment than optimizing any single component in isolation.

For developers, three actionable practice directions are: estimating cost from token counts, diagnosing anomalies from attention patterns, and tuning throughput through cache strategies. Making LLM mechanisms transparent ultimately serves more controllable and efficient model applications.

Note: This article is based on a Reddit community discussion; please refer to the corresponding model's technical documentation for specific implementation details.

One-line Summary

Breaking down tokenization, attention, and KV caching—three core mechanisms—to analyze LLM underlying logic and engineering trade-offs.

Paragraph Summary

Based on an in-depth Reddit community discussion, this article systematically breaks down three core mechanisms of large language models: tokenization splits text into subword units, and its strategy directly affects sequence length, context window capacity, and inference cost; self-attention uses Q/K/V computation to let tokens dynamically aggregate contextual information, but complexity grows quadratically with sequence length, spurring optimizations like sparsity and sliding windows; KV caching stores historical key-value vectors to avoid redundant computation in autoregressive generation, significantly reducing latency but bringing VRAM pressure that grows linearly with length, addressable through quantization, paged management, and selective discarding. The article emphasizes that understanding the interplay among these three is more effective for real-world deployment than optimizing any single component in isolation, and offers three practical directions: estimating cost via token counts, diagnosing via attention patterns, and tuning throughput via cache strategies.