Local LLM VRAM Budget: Weights, KV Cache, and Runtime Overhead

When deploying a local LLM, many people see a label like "8B, 4-bit" and assume that is the entire memory requirement. But what actually determines whether it runs is the sum of weights, conversation cache, and runtime overhead. This article uses a reproducible arithmetic framework to help you calculate all three parts clearly.

Weight Load: Starting from Parameter Count and Bit Width

The storage size of weights can be estimated with a simple formula: weight bytes = P × b / 8, where P is the number of scalar parameters and b is the number of bits per parameter.

Using a model with exactly 8 billion parameters as an example, the idealized loads are as follows:

• 16 bits: about 14.90 GiB
• 8 bits: about 7.45 GiB
• 4 bits: about 3.73 GiB

Note the difference between GiB (2^30 bytes) and decimal GB (10^9 bytes); downloads, memory monitors, and hardware specs often mix these two units.

More importantly, the numbers above assume all parameters use the same representation. Real quantized artifacts often include scaling factors, metadata, and tensors kept at higher precision. According to the Transformers bitsandbytes documentation, quantized linear layers and other modules are handled separately, and weight storage and compute dtype are two different things. So do not treat all "4-bit" labels as the same format; you must check the specific model artifact and runtime implementation.

Scaling Factors and Metadata in Quantization

4-bit quantization is not simply truncating each parameter to 4 bits. In bitsandbytes NF4 or GPTQ, for example, quantized weights are usually stored in blocks, each with a scaling factor (scale) and zero-point, and this metadata is often saved in FP16 or FP32. In addition, embedding layers, output layers, and normalization layers may remain at higher precision. As a result, the actual weight file is slightly larger than the theoretical value P×b/8, typically by 5%-15%.

Different quantization formats (GGUF, AWQ, GPTQ, bitsandbytes) have different metadata overhead and layouts, and peak memory during loading may also differ. When planning, use the actual file size in the model repository as the basis and leave a loading buffer.

KV Cache: The Longer the Conversation, the Faster It Grows

During autoregressive generation, the Transformer reuses previously computed keys and values, which is the KV cache. For conventional full-attention decoding layers, the formula for estimating the uncompressed cache is:

KV bytes = 2 × L × Hkv × D × T × C × S

where 2 represents keys and values, L is the number of attention layers, Hkv is the number of key/value heads per layer, D is the dimension of each key/value head, T is the number of tokens cached per sequence (including prompt and output), C is the number of bytes per cache element (2 for FP16/BF16), and S is the number of independently stored sequences.

Key point: use the number of KV heads, not the number of query heads. If sequence lengths differ, sum the token counts; the multiplication above assumes equal lengths and no shared prefix. Beam search or concurrent requests introduce more independently cached sequences.

Using the official Qwen3-8B configuration as an example: 36 layers, 8 KV heads, head dimension 128, sliding window attention disabled. With 8192 tokens cached, 2 bytes per element, and a single sequence:

2 × 36 × 8 × 128 × 8,192 × 2 × 1
= 1,207,959,552 bytes
= 1.125 GiB

Under the same assumptions, a single sequence of 4096 tokens is about 0.5625 GiB, a single sequence of 16384 tokens is about 2.25 GiB, and four sequences of 8192 tokens reach 4.5 GiB.

These are only tensor load estimates, not total memory usage, speed, or whether a particular machine can run it. Other architectures and cache implementations require adjusting the formula: sliding window layers may keep only a bounded window; static caches reserve up to the configured limit, while dynamic caches grow with use; cache quantization and offloading change memory location and precision and introduce performance trade-offs. Weight quantization does not automatically determine the KV cache dtype; check the behavior supported by the backend.

Grouped Query Attention (GQA) and KV Head Count

Modern LLMs (such as Qwen, Llama 3, Mistral, etc.) generally use Grouped Query Attention (GQA) rather than classic multi-head attention. In GQA, query heads are divided into groups, and each group shares a pair of key/value heads, so the number of key/value heads Hkv is usually much smaller than the number of query heads. This is why you must use Hkv rather than the number of query heads when estimating KV cache—using the wrong one will severely overestimate the result.

For Qwen3-8B, for example, it may have 32 query heads but only 8 KV heads, meaning the KV cache is only one quarter of that under traditional multi-head attention. This design significantly reduces memory pressure at long context while maintaining model quality, and it is also the mainstream choice for current open-source models. If your model uses multi-query attention (MQA), then Hkv=1, and the cache will be even smaller.

Execution and Loading: Space Left for the Runtime

The planning worksheet can be written as: resident weights + KV cache + runtime/workload overhead.

Check both loading peak and generation peak. According to the Accelerate documentation, the loading process may create additional weight copies; sharded loading can reduce this pressure, and CUDA initialization itself consumes memory beyond the weights. CPU or disk offloading changes memory location and introduces transfers; fitting in memory does not mean latency is acceptable.

For your specific runtime, check temporary workspace, prefill allocations, allocator reservations, and whether there are additional models or multimodal components. There is no universal overhead percentage; you must measure the specific workload and leave headroom.

On Apple chips, MLX uses unified memory shared by CPU and GPU. This shared capacity can only be counted once, and you must leave room for the operating system and other applications; do not count the same unified memory pool as both RAM and VRAM.

Unified Memory Architecture (Apple Silicon)

Apple M-series chips use a Unified Memory Architecture, where the CPU and GPU share the same physical memory pool and there is no separate VRAM. This means model weights, KV cache, operating system usage, and other application memory are all allocated from the same capacity. Therefore, when planning, this shared capacity can only be counted once; unlike discrete GPUs, you cannot add "system memory" and "VRAM" together.

The actually available capacity is usually less than the nominal value because macOS itself and background applications consume part of it. For example, on a machine with 32GB of unified memory, only about 24-28GB may actually be available for the model. When using frameworks such as MLX or llama.cpp, monitor memory pressure and reserve enough space for the system to avoid severe performance degradation caused by memory swapping.

Turning Estimates into a Reproducible Check

Record the following information: the exact repository, checkpoint revision number, and quantization artifact; runtime version and backend; cache dtype and policy; prompt/output token limits; number of concurrent sequences.

Start with a small load, then increase context and concurrency separately. Use the measurement tools supported by the runtime to record loading peak, generation peak, time to first token, and generation rate, and state how memory was measured—allocated memory or reserved memory.

This is a proposed measurement procedure; this article does not present any measured results or make claims such as "it runs easily on my laptop."

Finally, check the license for the exact version. The official Qwen3-8B repository used for the architecture example uses the Apache License 2.0, but do not extend this license to other Qwen versions, converted artifacts, or unrelated models. Record the applicable model and runtime licenses together with the revision number and deployment configuration.

A useful output is a budget with adjustable assumptions: shortening context lowers the conventional cache estimate; reducing weight precision changes the weight budget; adding independent sequences increases cache demand. Measurement tells you where the estimate begins to diverge from actual runtime.