Why Long Context Is Expensive: The Storage Bill of KV Cache

Long Context Isn't a Free Lunch

Your chat history might weigh more than the model that reads it. Llama 3.1 8B is a 16 GB model, but give it a 1 million token prompt, and the KV cache alone requires 128 GB. This contrast reveals an often-overlooked fact: the real bottleneck for long context capability isn't model parameters—it's the memory overhead during inference.

According to the Meta Llama 3.1 model card and unsloth configuration data, this figure isn't a theoretical estimate—it's the result of calculations based on actual deployment configurations. When we talk about "extending the context window to 1 million tokens," the real question we should be asking is: who pays for storing all those tokens?

Why KV Cache Exists

The Transformer architecture has no inherent memory. Early RNNs maintained a fixed-size hidden state—constant in size, but prone to forgetting early information, the classic vanishing gradient problem. Transformers take a completely different approach: they re-read the entire conversation history at every generation step.

To avoid recomputing this "re-reading" from scratch each time, every token is stored at every layer as Key and Value vectors. Store once, reuse forever. This storage structure is the KV cache. It keeps inference speed sustainable, but at the cost of linear growth in GPU memory usage.

According to Sebastian Raschka's analysis, understanding KV cache is key to understanding LLM inference costs—it's not an optional optimization, but an inevitable product of the Transformer architecture.

What KV Cache Is and What It Does

KV cache (Key-Value cache) is the data structure that stores the Key and Value vectors corresponding to every token at every layer during Transformer inference. In autoregressive generation, each new token requires computing attention weights with all historical tokens. Without caching historical Key/Value, every step would need to recompute the K, V matrices for the entire sequence, and computation would grow quadratically with sequence length. KV cache caches the already-computed K, V so that new tokens only need to compute attention with the cache, reducing per-step computation from O(n²) to O(n)—at the cost of GPU memory usage growing linearly with sequence length.

Vanishing Gradients and the RNN Comparison

Vanishing gradients refer to the exponential decay of gradients during backpropagation in RNNs due to repeated multiplication across time steps, making it hard for the model to learn long-range dependencies. This directly causes RNNs' hidden states to forget early information despite their fixed size. Transformers bypass this problem by connecting any two positions directly through self-attention—but at the cost of storing representations of all historical tokens.

KV Heads and Head Dim

In multi-head attention, each attention head independently computes K, V. KV heads refer to the number of heads for Key/Value projection, and head dim is the dimension per head. In standard multi-head attention, KV heads equal the number of query heads; GQA lets multiple query heads share a set of KV heads, thereby reducing cache size.

How the Storage Bill Is Calculated

The size of the KV cache can be expressed with a concise formula:

cache = 2 (K and V) × layers × KV heads × head dim × tokens × 2 bytes

For Llama 3.1 8B, this formula yields 128 KiB per token. That doesn't sound like much, but at scale the numbers become staggering:

• 128K context: 16 GB—the same size as the model itself
• 1 million tokens: 128 GB—exceeding the memory capacity of a single H100

Larger models pay more. The 70B version needs 320 KiB per token, taking 43 GB at 128K context and soaring to 344 GB at 1 million tokens. According to spheron's KV cache guide, Grouped Query Attention (GQA) keeps this bill within acceptable range by reducing KV heads from 32 to 8 shared heads—otherwise these numbers would be 4 to 8 times worse.

Deriving and Interpreting the Formula

In 2 (K and V) × layers × KV heads × head dim × tokens × 2 bytes, the first 2 indicates storing both Key and Value tensors; layers is the number of Transformer layers; KV heads × head dim is the total dimension of the K or V vector per token per layer; tokens is the sequence length; and the final 2 bytes is the number of bytes per parameter under FP16 (half precision). With FP32, this value doubles; with INT8 quantization, it halves.

GQA (Grouped Query Attention)

GQA groups query heads, with each group sharing a single Key/Value head. For example, Llama 3.1 8B has 32 query heads but only 8 KV heads, shrinking the KV cache to 1/4 of its original size. Compared to MQA (Multi-Query Attention), where all heads share one set of KV, GQA achieves a better balance between quality and efficiency.

H100 Memory and Context

NVIDIA H100 is the mainstream AI training/inference GPU today, commonly available in an 80 GB memory version. A 1 million token KV cache requires 128 GB, meaning a single H100 cannot accommodate it—multiple cards or offloading techniques are required. This vividly illustrates the hardware barrier for long context.

The Quadratic Overlay of Attention

Memory is only part of the bill. The computational complexity of the attention mechanism grows quadratically with sequence length, meaning longer contexts not only consume more GPU memory but also increase the processing cost per word.

This dual pressure—linear memory growth plus quadratic computational growth—explains why long context inference is so expensive in real-world deployment. Based on cross-validation of multiple sources above, this conclusion is consistently reflected in the Meta model card, unsloth configuration, and spheron guide.

Where Quadratic Complexity Comes From

In self-attention computation, each token's query vector must be dot-producted with all tokens' key vectors, producing an n×n attention score matrix, so computation is O(n²·d), where n is sequence length and d is dimension. When n grows from 128K to 1M, computation increases roughly 64×, while KV cache grows only about 8×. This means even with sufficient GPU memory, computation time becomes the bottleneck.

Common Mitigation Methods

FlashAttention reduces memory reads/writes through block-wise computation, improving actual speed without changing theoretical complexity; sparse attention and sliding window attention (e.g., Mistral) compute only local or partial attention, reducing complexity to O(n) or O(n log n); linear attention approximates softmax attention with kernel functions. Each method has trade-offs and none has yet fully replaced standard attention.

Long Context Is a Storage Cost Problem

Long context isn't a memory upgrade—it's a storage bill. At a certain scale, the conversation itself becomes the most expensive object in the data center.

This shift in perspective matters: when evaluating a model's long context capability, we shouldn't just look at how many tokens it supports, but also at the storage cost per token. Architectural optimizations like GQA can mitigate the problem, but cannot change the fundamental linear growth. For teams building long context applications, KV cache budget planning is as important as model selection.

One-line Summary

The real bottleneck of long context is the linear storage cost of KV cache plus the quadratic computational overhead of attention.

Paragraph Summary

The article points out that the core bottleneck of long context capability isn't model parameters, but the KV cache memory overhead during inference. Taking Llama 3.1 8B as an example, the model itself is only 16 GB, but the KV cache for a 1 million token context reaches 128 GB, far exceeding a single H100's memory. KV cache is an inevitable product of Transformer's avoidance of redundant computation, with its size determined by the number of layers, KV heads, head dimension, and token count—costing 128 KiB per token; the 70B model reaches 320 KiB, requiring 344 GB for a 1M context. GQA reduces overhead by 4-8× by sharing KV heads. Meanwhile, the quadratic complexity of attention computation causes long context costs to rise on two fronts. The article concludes: long context is essentially a storage bill, and KV cache budget planning should be as important as model selection.