Why Does a 35GB Quantized Model Blow Up an 80GB GPU? The Truth About KV Cache VRAM

Quantize a 70B model to 4-bit, and by the rough estimate of "0.5 bytes/parameter" you'd only need about 35GB of VRAM—theoretically more than enough to fit on a single 80GB A100/H100. But when you actually fire up vLLM or llama.cpp and open a 64K context window, torch.OutOfMemoryError: CUDA out of memory shows up right on schedule.

The root of the problem: model weights are merely the static floor of LLM VRAM usage. The real heavyweight in VRAM is the KV Cache, followed by quantization scale overhead, prefill activation buffers, and CUDA runtime reservations. Add these together, and actual usage can be two to three times the weights.

The Four Components of Inference VRAM

Runtime GPU VRAM consists of four independent allocations:

Total VRAM = M_weights + M_kv_cache + M_activations + M_runtime_overhead

Component	Scaling Factor	Share at Short Context	Share at 128K Context/Batch16
Model weights	Parameters × effective bit width	75%–85%	30%–45%
KV Cache	Batch × context length × layers	5%–15%	45%–65%
Activation buffers	Batch × Prefill chunk × hidden dim	3%–8%	5%–10%
CUDA/runtime overhead	Fixed value + PagedAttention block table	1.5–3.0 GB	2.0–4.0 GB

The longer the context and the higher the concurrency, the more terrifying the KV Cache share becomes. At 128K context with batch 16, it already consumes nearly two-thirds of VRAM. This explains why "the weights fit" and "it actually runs" are two entirely different things.

4-bit Quantization Is Never Exactly 0.5 Bytes

The rule of thumb "INT4 means 0.5 bytes/parameter" doesn't hold in production. Weight distributions contain extreme outliers, and naive uniform quantization directly destroys model perplexity.

Mainstream quantization formats (AWQ, GPTQ, GGUF K-Quants) work by grouping weights (typical group_size of 32/64/128), storing one high-precision FP16 scale and zero-point per group, while keeping embedding and lm_head layers at 16-bit or 8-bit precision. The actual effective bits per weight (bpw) is far higher than 4:

• FP16 / BF16: 16.0 bpw (2.0 bytes/parameter)
• FP8 / INT8: ~8.5 bpw (including scale factors, ~1.06 bytes/parameter)
• INT4 AWQ (group_size=128): ~4.5 bpw (~0.5625 bytes/parameter)
• GGUF Q5_K_M: ~5.7 bpw (~0.71 bytes/parameter)
• GGUF Q4_K_M: ~4.85 bpw (~0.606 bytes/parameter)

Taking a 70.6B model in GGUF Q4_K_M format as an example, the weights alone come to about 39.8 GiB—nearly 5GB more than the off-the-cuff 35GB. That 5GB looks insignificant, but combined with the KV Cache discussed below, it's the last straw that breaks the 80GB camel's back.

The Silent Killer: The Precise Math of KV Cache

During autoregressive generation, if you recompute the Key and Value of every historical token at every step, decoding complexity degrades to O(N²). So inference engines cache the K and V tensors for each layer. For models using Grouped Query Attention (GQA) such as the Llama, Qwen, and Mistral families, the KV Cache byte count is:

KV_Bytes = 2 (K and V)
         × n_layers
         × n_kv_heads
         × head_dim
         × seq_len
         × batch_size
         × bytes_per_kv_element

Plugging in the real architectural constants of a 70B GQA model (n_layers=80, n_kv_heads=8, head_dim=128, FP16 cache = 2 bytes):

• Per-token cache: 2 × 80 × 8 × 128 × 2 = 327,680 bytes ≈ 320 KB/token
• 64K context × batch=2: 320 KB × 65,536 × 2 ≈ 39.06 GiB

Now let's add up the tab:

• Weights (Q4_K_M): 39.8 GiB
• KV Cache (64K × batch2, FP16): 39.1 GiB
• Activations + CUDA runtime: ~4.2 GiB
• Total: ~83.1 GiB → instant OOM on an 80GB A100/H100

Key comparison: If this model used classic Multi-Head Attention (MHA, 64 KV heads) instead of 8-head GQA, the same KV Cache would balloon to 312.5 GiB—no longer a question of fitting on a single card. GQA exists precisely to bring the KV Cache down from catastrophic to manageable, but even after that compression, long contexts remain ferocious.

Grouped Query Attention (GQA) is a compromise attention architecture between MHA and MQA. In classic MHA, every attention head has independent Query, Key, Value projections, so the number of KV heads equals the number of Query heads (e.g., 64 KV heads in Llama-2 70B); in MQA, all Query heads share a single set of K and V, reducing the KV head count to 1. GQA divides Query heads into several groups, each sharing one set of K and V, with a typical configuration of 8 KV heads (every 8 Query heads share one set). Since KV Cache size is proportional to n_kv_heads, GQA compresses the KV Cache by roughly 8× relative to MHA while retaining better modeling capability than MQA—hence its adoption as the standard choice for mainstream open-source models like Llama 2/3, Qwen, and Mistral.

How DeepSeek MLA Rewrites This Curve

DeepSeek^Ad-V3 and DeepSeek-R1 use Multi-head Latent Attention (MLA). Their total parameter count is a 671B MoE (37B activated). If you expanded 128K context across 61 layers with standard GQA, the PCIe/NVLink bandwidth for the KV Cache alone would become the bottleneck.

MLA's approach is to jointly project Key and Value into a compressed low-rank latent vector (c_t^KV, typical dimension d_c=512), plus an attached decoupled RoPE key dimension (d_R=64). What gets cached per layer is no longer 2 × n_kv_heads × head_dim elements, but only 512 + 64 = 576 elements:

if (arch.attentionType === 'MLA') {
  const compressedDim = (arch.kvLoraRank ?? 512) + (arch.qkRopeHeadDim ?? 64);
  // MLA caches only the joint latent vector per layer, with no 2× multiplier for K/V
  return arch.numLayers * compressedDim * arch.kvBytesPerElem;
}

Note there's no 2× multiplier like in the GQA formula, and no n_kv_heads term. The result is a roughly 70%–80% reduction in KV Cache memory compared to an equivalent GQA model. According to the consensus of multiple independent technical analyses, this is not a marketing number but a direct memory benefit of MLA's architectural design—and one of the key reasons DeepSeek can support long contexts on a single node of H800s.

Multi-head Latent Attention (MLA) differs from GQA primarily in the abstraction level of what gets cached: GQA still caches full K and V tensors per KV head, just with fewer heads; MLA jointly projects each layer's Key and Value into a low-rank latent vector (typical dimension 512), caching only this compressed vector plus a decoupled RoPE dimension (64) at inference time, then restoring K and V via up-projection matrices when needed. This reduces the per-layer cache unit from 2 × n_kv_heads × head_dim (typically 2×8×128=2048 for GQA) to d_c + d_R (576), with no 2× K/V multiplier. Note that MLA's extra compute overhead comes from the up-projection matrices—a classic "trade compute for memory" design. The gains are significant in long-context scenarios, though it may be less efficient than GQA for short sequences.

Three Production Levers to Avoid OOM

Even if the architecture isn't in your hands, there are clear VRAM reduction measures on the deployment side:

1. Quantize the KV Cache to FP8 (--kv-cache-dtype fp8 in vLLM). This drops bytes per element from 2 (FP16) to 1 (FP8 E4M3), directly halving KV Cache usage. On Hopper/Ada GPUs, the accuracy loss is nearly negligible.

2. Cap --max-model-len based on your actual P99 request length. Just because a model supports 128K context doesn't mean vLLM should pre-allocate block tables for 128K. If your P99 request is under 16K, capping the limit at 16K frees up substantial space.

3. Enable --enable-chunked-prefill. Without chunked prefill, a single 64K token prompt causes intermediate MLP activation tensors to spike during the prefill phase. Chunked prefill constrains peak activation memory to a fixed per-step token budget.

Used together, these three tactics—combined with architectural improvements like MLA—are the complete answer for keeping VRAM stable under long-context, high-concurrency inference. VRAM planning isn't as simple as "the weights fit"—you need to treat the KV Cache as a first-class citizen in your calculations.

What GPU configuration are you currently using for local or production inference? Have you tested FP8 KV Cache quantization under long-context workloads? Feel free to share in the comments.

Chunked prefill works by splitting a long prompt into multiple fixed-size token chunks (e.g., 512 or 1024 tokens), executing prefill chunk by chunk while scheduling them alongside decode requests, rather than stuffing the entire prompt into a forward pass at once. The benefits are twofold: first, it reduces peak activation memory during prefill from O(prompt length) to O(chunk size), preventing a 64K prompt from instantly blowing up VRAM; second, it stops long prefills from blocking decode requests, significantly reducing TTFT and ITL jitter. vLLM has supported this feature since version 0.4, but when enabled, the single-step computation graph contains both prefill and decode workloads, which may impose additional compatibility requirements on certain custom kernels—so throughput regression comparisons are needed before production deployment.

One-line Summary

Inference VRAM consists of weights, KV Cache, activations, and runtime—and under long contexts, the KV Cache is the real OOM killer.

Paragraph Summary

The article corrects the common misconception that "a 4-bit quantized 70B model only needs 35GB of VRAM." Real VRAM usage comprises four parts: model weights, KV Cache, activation buffers, and CUDA runtime—with weights being merely the static floor. 4-bit quantization actually has an effective bit width of about 4.5–4.85 bpw, making a 70B model's weights roughly 39.8GiB in practice. Under long contexts and high concurrency, the KV Cache can consume 45%–65% of VRAM. A 70B GQA model at 64K context, batch=2, with FP16 caching hits 39GiB for KV Cache alone, bringing total demand to about 83GiB and causing immediate OOM. DeepSeek's MLA architecture compresses per-layer KV caching to 576 elements, cutting VRAM by about 70%–80% versus GQA. Production environments can prevent OOM through three measures: FP8 KV Cache quantization, limiting max-model-len, and enabling chunked prefill.