How Much VRAM Does a 70B LLM Need? Full Breakdown of Inference and Fine-Tuning

Deploying or fine-tuning a 70B-parameter model—how much VRAM does it actually take? This is the core question every team trying to ship an LLM application runs into. The bottom line up front: serving a 70B model at 16-bit precision needs about 141 GB of VRAM for the weights, FP8 cuts that to 73 GB, and 4-bit quantization needs just 40 GB; on top of that, every 1,000 in-flight tokens require roughly 0.33 GB of KV cache. For fine-tuning, QLoRA needs about 50 GB, LoRA about 150 GB, and full fine-tuning with the AdamW optimizer balloons to 1.1 TB.

These numbers come from the Llama 3.3 70B config file and real checkpoint sizes on Hugging Face, and the calculations for Qwen2.5 72B come out nearly identical.

Where Does VRAM Actually Go During Inference?

GPU VRAM is consumed by three things: model weights, the KV cache, and the working headroom reserved by the CUDA context and the inference engine itself.

Weights are the most intuitive part—however big the checkpoint is on disk, that's how big it is loaded into memory. Here are three public versions of the same model:

Precision	Hugging Face checkpoint	Weights in memory
16-bit (BF16)	meta-llama/Llama-3.3-70B-Instruct	141.1 GB
8-bit (FP8)	RedHatAI/Llama-3.3-70B-Instruct-FP8-dynamic	72.7 GB
4-bit (AWQ INT4)	casperhansen/llama-3.3-70b-instruct-awq	39.8 GB

The actual size of quantized versions runs a few GB above the "parameter count × bits" calculation, because the embedding and output layers stay at 16-bit precision. Together those two layers hold 2.1 billion parameters, accounting for 4.2 GB, plus the 4-bit format also needs to store scaling factors.

As for the working headroom, vLLM by default plans around 90% of GPU VRAM via --gpu-memory-utilization, and only the remainder is used as KV cache—so the KV cache figures below should be treated as an upper bound.

How Big Is the KV Cache, Really?

Each token needs about 0.33 MB, and it grows linearly with the number of concurrent requests. The formula is: 2 (keys and values) × layers × KV heads × head dimension × 2 bytes. Llama 3.3 70B has 80 layers, 8 KV heads, and a head dimension of 128, so each token consumes 2×80×8×128×2 = 327,680 bytes. Qwen2.5 72B has the same structure and the same overhead.

In real traffic, here's how that plays out:

• A single 8,192-token conversation occupies 2.7 GB of cache
• Handling 32 such conversations simultaneously requires 85.9 GB—more than double the 4-bit weights
• A single request using the full 131,072-token context window needs 43 GB

The 8-KV-head design is critical here. Llama 3 uses grouped query attention, where 64 query heads share 8 KV heads. If each query head had its own KV head, the cache would balloon 8× to about 2.6 MB per token.

The common rule of thumb is to add 20% on top of the weights to cover cache and other overhead. For 35 GB of 4-bit weights that's about 7 GB, which fits only about 21,000 tokens—enough for a few conversations, but not enough to sustain a busy serving endpoint. A more sensible approach is to calculate from actual traffic: peak concurrent requests × (input + output tokens) × 0.33 MB.

vLLM supports storing the cache in FP8 via --kv-cache-dtype fp8, halving the footprint to about 0.16 MB per token, but it's best to validate quality on your own eval set first.

Grouped Query Attention (GQA) is the attention variant adopted by default in modern LLMs like Llama 3 and Qwen2.5. In standard multi-head attention (MHA), every query head has its own dedicated key head and value head, so the KV cache scales with the number of query heads; GQA groups multiple query heads to share a single set of KV heads, dramatically shrinking the cache. Llama 3.3 70B sets 64 query heads and 8 KV heads, meaning 8 query heads share 1 KV head, which cuts the KV cache to 1/8 of MHA. This design brings the VRAM cost of long-context and high-concurrency scenarios down to an acceptable range with virtually no loss in model quality. For comparison, multi-query attention (MQA) has all query heads share a single KV head—more aggressive compression but with more noticeable quality loss; GQA is the compromise between quality and efficiency, and has been widely adopted by mainstream models like Llama 2 70B, the entire Llama 3 line, Mistral, and Qwen2.

Multi-head Latent Attention (MLA) is another attention structure proposed by the DeepSeek^Ad model family. It first projects keys and values into a low-dimensional latent vector before caching, further reducing per-token KV overhead to a fraction of GQA. Because MLA's KV shape is completely different from GQA's, the "0.33 MB per token" formula in this article does not apply to MLA models—you'd need to derive it separately based on the latent dimension and layer count. When selecting a model, if the target uses MLA, be sure to check fields like kv_lora_rank in the official config file and recalculate.

Which GPU Configurations Can Serve a 70B Model?

Any configuration works as long as the weights fit within 90% of total VRAM and leave room for cache. Here are options worth considering:

Configuration	Weights	KV cache available	Max in-flight tokens	Price per hour
1×A100 80GB, 4-bit	39.8 GB	32.2 GB	~98,000	$1.09
1×RTX PRO 6000 96GB, 4-bit	39.8 GB	46.6 GB	~142,000	$1.35
1×H200 141GB, FP8	72.7 GB	54.2 GB	~165,000	$3.59
2×H100 80GB, FP8	72.7 GB	71.3 GB	~218,000	$5.20
1×B200 180GB, FP8	72.7 GB	89.3 GB	~272,000	$5.50
1×B300 288GB, 16-bit	141.1 GB	118.1 GB	~360,000	$6.94
4×H100 80GB, 16-bit	141.1 GB	146.9 GB	~448,000	$10.40

Two configurations look viable on paper but fail in practice: 2×80GB GPUs running 16-bit precision, which leaves only 2.9 GB after the weights load—about 9,000 tokens across all requests; and 2×24GB GPUs running 4-bit precision, which leaves only 3.4 GB, about 10,000 tokens. Both can load the model, but there's almost no headroom to actually serve requests.

Precision choice also constrains hardware. FP8 is natively supported on Hopper and Blackwell architecture GPUs (H100, H200, B200, B300), but the A100 has no FP8 hardware support and can only use 4-bit checkpoints.

Selection rule: first pick the precision based on quality evaluations, then multiply peak concurrent requests by total tokens and 0.33 MB, and choose the cheapest configuration that meets the requirement. If a single request needs the full 128K context, you'll need at least 43 GB of cache beyond the weights.

How Much VRAM Does Fine-Tuning a 70B Model Need?

Far more than inference, because training needs to keep gradients and optimizer states for every updated parameter. The ZeRO paper notes that mixed-precision training with the Adam optimizer requires 16 bytes per parameter: 2 bytes for the 16-bit weights, 2 bytes for gradients, and 12 bytes for the FP32 master copy plus Adam's two moment estimates.

• Full fine-tuning: 70.55 billion parameters × 16 bytes = 1,129 GB. bitsandbytes' 8-bit optimizer compresses each moment estimate from 4 bytes to 1 byte, bringing it down to 10 bytes per parameter, or 706 GB.
• LoRA: the frozen 16-bit base model takes 141.1 GB, and adding rank-16 adapters to each linear layer yields about 207 million trainable parameters (0.3% of the model), at 16 bytes per parameter with FP32 + Adam, totaling 3.3 GB.
• QLoRA: the base model is stored in 4-bit NF4, with linear layers taking 34.2 GB, double-quantization scaling constants about 1.1 GB, and 16-bit embeddings and output layers 4.2 GB, for a total of 39.5 GB, plus the same 3.3 GB of adapters.

Activations are counted separately and depend on sequence length. With gradient checkpointing, only the input is kept per layer: 4,096 tokens × 8,192 hidden dimension × 2 bytes × 80 layers = 5.4 GB per 4,096-token sequence. FP32 logits over a 128,256-token vocabulary add another ~2.1 GB. Estimate roughly 7.5 GB per sequence per GPU.

Method	Before activations	Including one 4,096-token sequence	Minimum viable configuration	Price per hour
QLoRA	43 GB	~50 GB	1×A100 80GB / 1×H100 / 1×RTX PRO 6000	from $1.09
LoRA	144 GB	~152 GB	1×B200 / 1×B300 / 2×H200 / 4×H100	from $5.50
Full + 8-bit Adam	706 GB	~766 GB on 8 GPUs	8×H200 (1,128 GB)	$28.74
Full + AdamW	1,129 GB	~1,189 GB on 8 GPUs	8×B200 (1,440 GB)	from $44.00

Crossing multiple GPUs requires a sharding scheme like PyTorch FSDP or DeepSpeed ZeRO-3, so each GPU holds only a fraction of the weights, gradients, and optimizer states. Two 8×H100 machines offer 1,280 GB total, leaving only about 9 GB per GPU under full AdamW training—too tight for most scenarios.

ZeRO (Zero Redundancy Optimizer) is a VRAM optimization technique from DeepSpeed. Its core idea is to shard training state (weights, gradients, optimizer states) across multiple GPUs, avoiding a full copy on every card. ZeRO has three stages: Stage 1 shards only optimizer states, Stage 2 additionally shards gradients, and Stage 3 further shards model weights. The "16 bytes per parameter" estimate in this article corresponds to the single-card footprint when unsharded; in real multi-GPU training, Stage 2 or Stage 3 distributes optimizer states and gradients across cards, which is why 8×H200 can accommodate the 706 GB full fine-tuning requirement. PyTorch's native FSDP (Fully Sharded Data Parallel) implements functionality equivalent to ZeRO-3.

Note that sharding doesn't reduce total VRAM needs—it just spreads the total across more GPUs. Additionally, sharding introduces extra communication overhead: after each forward and backward pass, cards need to synchronize gradients or re-aggregate weight shards, so ZeRO-3's communication overhead rises significantly as GPU count increases, and real training throughput may not scale linearly with card count. For 70B full fine-tuning, 8 GPUs is the minimum viable scale; better throughput usually requires combining tensor parallelism (TP) and pipeline parallelism (PP) into a 3D parallel strategy.

Gradient Checkpointing is another key VRAM technique: during training, only the activations of some layers are retained; the rest are not stored during the forward pass and are recomputed on demand during the backward pass. This trades roughly 30% more compute for reducing activation VRAM from scaling with layer count to nearly constant. The "only the input is kept per layer" estimate in this article is based on gradient checkpointing.

Can You Fine-Tune a 70B Model on a Single GPU?

Yes—with QLoRA, any GPU with 80 GB or more works. The QLoRA paper famously fine-tuned a 65B model on a single 48 GB GPU. Llama 3 70B needs about 43 GB before activations, leaving only about 5 GB on a 48 GB card (like the L40S or RTX A6000)—enough for short sequences only.

LoRA on a 16-bit base is only feasible on a single B200 or B300. There is no single-GPU option for full fine-tuning a 70B model.

How Do You Verify the Math for Your Own Model?

Just run the same formulas against your target model's config file. The script below reads config.json and the parameter count from the Hugging Face API, and outputs the estimates above using only the Python standard library:

"""Rough GPU memory estimate for serving and fine-tuning a Hugging Face model."""
import json, os, sys, urllib.request

def get(url):
    req = urllib.request.Request(url)
    if os.environ.get("HF_TOKEN"):
        req.add_header("Authorization", f"Bearer {os.environ['HF_TOKEN']}")
    with urllib.request.urlopen(req) as r:
        return json.load(r)

repo = sys.argv[1]
context = int(sys.argv[2]) if len(sys.argv) > 2 else 8192
seqs = int(sys.argv[3]) if len(sys.argv) > 3 else 1
cfg = get(f"https://huggingface.co/{repo}/resolve/main/config.json")
params = get(f"https://huggingface.co/api/models/{repo}")["safetensors"]["total"]

head_dim = cfg.get("head_dim") or cfg["hidden_size"] // cfg["num_attention_heads"]
kv_heads = cfg.get("num_key_value_heads", cfg["num_attention_heads"])
kv_per_token = 2 * cfg["num_hidden_layers"] * kv_heads * head_dim * 2
GB = 1e9

print(f"{repo}: {params / 1e9:.1f}B parameters")
for name, bytes_per_param in [("16-bit", 2), ("8-bit", 1), ("4-bit", 0.5)]:
    print(f"  weights at {name}: {params * bytes_per_param / GB:6.1f} GB")
print(f"  KV cache: {kv_per_token / 1e6:.3f} MB per token, "
      f"{kv_per_token * context * seqs / GB:.1f} GB for {seqs} x {context:,} tokens")
print(f"  full fine-tune, AdamW mixed precision (16 B/param): {params * 16 / GB:,.0f} GB + activations")
print(f"  LoRA on a 16-bit base: {params * 2 / GB:,.0f} GB + adapters + activations")
print(f"  QLoRA on a 4-bit base: {params * 0.5 / GB:,.0f} GB + a few GB + adapters + activations")

Sample output for Llama 3.3 70B with 32 conversations of 8,192 tokens:

unsloth/Llama-3.3-70B-Instruct: 70.6B parameters
  weights at 16-bit:  141.1 GB
  weights at 8-bit:   70.6 GB
  weights at 4-bit:   35.3 GB
  KV cache: 0.328 MB per token, 85.9 GB for 32 x 8,192 tokens
  full fine-tune, AdamW mixed precision: 1,129 GB + activations
  LoRA on a 16-bit base: 141 GB + adapters + activations
  QLoRA on a 4-bit base: 35 GB + a few GB + adapters + activations

The 8-bit and 4-bit rows are pure parameter calculations; actual checkpoints need a few extra GB. The KV formula assumes standard grouped query attention, and models with different designs like multi-head latent attention need their own formulas.

FAQ

Why do the weights fit but vLLM still reports out of memory?

At startup, vLLM checks whether the KV cache can hold at least one request of the model's maximum length. Llama 3.3 70B defaults to 131,072 tokens, which needs 43 GB, so a tight configuration will refuse to start. Lower --max-model-len to the actual maximum request length you serve, or use FP8 KV cache to free up space.

Do mixture-of-experts models work the same way?

For weights, calculate by total parameter count, not activated parameter count. Every expert must reside in GPU VRAM, even if only a few are activated per token, so a MoE model with 70B total parameters needs the same weight memory as a dense 70B.

Can a 70B model run on a single 24GB or 32GB GPU?

Not entirely in GPU VRAM—even 4-bit weights are about 40 GB. Tools like llama.cpp can place some layers in CPU memory, but those layers run at the mercy of CPU memory bandwidth.

How much disk space does fine-tuning a 70B model need?

16-bit weights take 141 GB. A resumable full fine-tuning checkpoint also needs the FP32 master weights and two Adam moments, roughly 12 bytes per parameter, adding another ~850 GB.

Differences between quantization formats: BF16 is 16-bit floating point, preserving full dynamic range, and is the default precision for training and inference; FP8 is 8-bit floating point with hardware acceleration on Hopper and Blackwell architectures and usually minimal quality loss; INT8/INT4 are integer quantizations that need scaling factors to convert integers back to floats—AWQ, GPTQ, and bitsandbytes NF4 all fall into this category. 4-bit NF4 (Normal Float 4) is the information-theoretically optimal 4-bit data type proposed in the QLoRA paper, designed specifically for normally distributed weights, which is why it's widely adopted in fine-tuning scenarios.

--gpu-memory-utilization in vLLM controls the fraction of GPU VRAM the engine may occupy, defaulting to 0.9. This value includes all overheads—weights, activations, CUDA context, etc.—and only the remainder is allocated as KV cache. So the "KV cache available" column in this article is effectively 0.9× total VRAM minus the weights. Lowering this value (e.g., to 0.85) leaves room for other processes but reduces the number of concurrent tokens you can serve.

PageAttention is vLLM's core technology, borrowing the idea of virtual memory paging from operating systems. It splits the KV cache into fixed-size blocks allocated on demand, eliminating the VRAM waste caused by reserving maximum length in traditional implementations and driving fragmentation down to near zero. This is also why vLLM can serve more concurrent requests within the same VRAM.

One-line Summary

70B model inference needs 141GB of weights at 16-bit and just 40GB at 4-bit; fine-tuning ranges from 50GB for QLoRA to 1.1TB for full AdamW.

Paragraph Summary

This article systematically breaks down the VRAM requirements for 70B-parameter LLMs across inference and fine-tuning. On the inference side, VRAM is split among weights, KV cache, and working headroom: 16-bit weights take about 141 GB, FP8 about 73 GB, and 4-bit AWQ just about 40 GB; the KV cache runs at 0.33 MB per token and grows linearly with concurrency, with 32 concurrent 8K conversations occupying 85.9 GB—more than the 4-bit weights themselves. The article presents seven viable GPU configurations from a single A100 up to 4×H100, and notes that the A100 lacks FP8 hardware acceleration. On the fine-tuning side, QLoRA at about 50 GB runs on a single 80GB GPU, LoRA at about 152 GB requires a B200 or B300, and full fine-tuning with AdamW reaches 1.1 TB, requiring an 8×B200-class cluster. A ready-to-run Python verification script is provided at the end.