KV Cache VRAM Math: Why Your Model Fits but Your Context Doesn't

A Counterintuitive VRAM Puzzle

Anyone who has deployed a large model locally has probably run into this: a quantized 8B model takes up about 4.9 GB on disk with Q4_K_M — theoretically, a 12 GB GPU could fit two copies with room to spare. Yet when you stretch the context window, it OOMs (out-of-memory) without mercy.

This isn't a GPU problem, and it's not a quantization issue. It's the hidden cost that almost nobody mentions on the download page: KV Cache. This memory overhead grows linearly with context length, and in long-context scenarios it can easily exceed the model weights themselves.

This article breaks down the memory math that nobody shows you, so you can calculate your hardware's real limits in under a minute.

The Core Formula for KV Cache VRAM

Breaking Down the Formula

The KV cache calculation is actually straightforward:

KV bytes = 2 × num_layers × num_kv_heads × head_dim × context_tokens × bytes_per_value

Let's unpack each parameter:

• 2: represents the Key and Value caches
• bytes_per_value: 2 bytes for fp16 precision
• num_hidden_layers and num_key_value_heads: these are the two variables that actually determine cache size

Working Through Llama 3.1 8B

Take Llama 3.1 8B as a concrete example. Its actual configuration is: 32 layers, 8 KV heads (note: it uses GQA — Grouped Query Attention — not 32 heads), and a head_dim of 128.

Plugging into the formula to get the cache cost per token:

2 × 32 × 8 × 128 = 65,536 values/token
65,536 × 2 bytes = 128 KB/token

The cache cost at different context lengths becomes immediately clear:

Context Length	KV Cache Size
2K	256 MB
8K	1 GB
32K	4 GB
128K	16 GB

The model weights are 4.9 GB, but at 128K context, the KV cache hits 16 GB — more than three times the model itself. A 12 GB GPU obviously can't handle that.

GQA (Grouped Query Attention) is the key prerequisite for understanding this result. In traditional Multi-Head Attention (MHA), every query head has its own dedicated Key and Value heads — a model with 32 attention heads also has 32 KV heads. GQA lets multiple query heads share a single set of KV heads: Llama 3.1 8B has 32 query heads but only 8 KV heads, meaning every 4 query heads share one KV group. This design compresses KV cache size to one-quarter of MHA without meaningfully degrading model quality. MQA (Multi-Query Attention) is the more extreme variant — all query heads share a single KV pair — which produces an even smaller cache but with greater quality trade-offs. GQA has become the standard in mainstream open-source models (Llama, Mistral, Qwen, and similar families) precisely because it strikes a solid balance between inference efficiency and model performance.

Why You Might Not Have Hit This Problem Yet

The answer is buried in the default settings of your inference framework. Ollama, for instance, does not use the model's full advertised context window by default. If you've never manually adjusted the num_ctx parameter, you've been running with a very small window, and KV cache overhead is practically negligible.

The problem surfaces the moment you see "supports 128K context" on a model card and enthusiastically max it out:

/set parameter num_ctx 32768

The cache cost appears instantly — and hits hard — because it scales linearly with token count. Many people assume the model somehow "got heavier." In reality, they just opened a bill that was always waiting.

The Metrics That Actually Matter When Choosing a Model: KV Heads and Layers

This is probably the most valuable point in this entire article: two models with identical parameter counts can have wildly different KV cache costs.

The reason lies in the difference between kv_heads and layers:

• A model with 8 KV heads uses only one-quarter the cache of an equivalent model with 32 KV heads
• Legacy Multi-Head Attention (MHA) models are extremely VRAM-hungry at long contexts
• GQA models are dramatically cheaper

The catch: none of this is visible in the parameter count or the quantization name. You have to open the model's config.json yourself and read the num_key_value_heads and num_hidden_layers fields.

The frustrating part is that these two numbers almost never appear on the same page as the download button. But ultimately, a config.json and a calculator will get you to the same answer.

Beyond num_key_value_heads and num_hidden_layers, head_dim (the dimension per head) in config.json also affects cache size — it's typically equal to hidden_size / num_attention_heads. On Hugging Face model pages, some models provide a config.json preview directly under the "Model card" or "Files" tab, no download required. You can also run ollama show <model_name> to inspect key parameters of any model you've already pulled with Ollama — it lists fields like attention heads and layers, making it a convenient shortcut for quick KV cache estimates.

A Practical Rule of Thumb: How to Budget Your VRAM

Here's a principle you can use directly:

VRAM budget = model weights + KV cache. Never budget for weights alone.

On consumer hardware, a smaller GQA model with a large context window will typically outperform a larger model locked into a 4K window. For many real-world tasks, fitting more context is worth more than a few extra billion parameters.

Next time you're deploying a model locally and trying to choose between options, take a minute to run the KV cache multiplication first. It'll save you from an embarrassing OOM — and help you put your limited VRAM exactly where it counts.