Why Your Computer Can't Run AI Models: Video Memory Is the Real Barrier

Many people download open-source AI models only to find they can't run them at all—the disk space is clearly sufficient, so why does it still fail? The answer lies in a commonly misunderstood concept: model parameters are not the same as file size, and running a model requires high-speed video memory that the GPU can directly access.

Parameters Are Not GB: Understanding the True Cost of Model Storage

A model labeled "8B" refers to 8 billion parameters, not 8GB. Each parameter is essentially a number, and how much space is needed to store these numbers depends on how many bits you use to store each one.

When stored at 16-bit precision (FP16), 8 billion parameters occupy approximately 16GB of space. This is the default precision for most model releases and the most intuitive source of file size. But 16GB is only the starting point—loading a model also requires additional overhead, including the inference framework's own footprint, KV cache (used to store conversation context), and more. So an 8B model often consumes 17GB or even more of video memory during actual operation.

Precision (FP16, FP32, INT4, etc.) refers to the number of bits used to store a single parameter. FP16 means each parameter is represented by a 16-bit (2-byte) floating-point number, while FP32 uses 32 bits (4 bytes). The larger the bit width, the higher the range and precision of values it can represent, but the space required also multiplies. Models are often trained with FP32 or mixed precision, but inference releases are typically converted to FP16 to halve the size. So-called 4-bit quantization (INT4) compresses each parameter to just 4 bits, making the size only one quarter of FP16, at the cost of reduced numerical precision, which may lead to some loss in output quality. Understanding the conversion relationship between bit width and size (parameter count × bit width ÷ 8 = bytes) lets you quickly estimate the storage and video memory floor for any model.

Quantization Saves Space, But Not the "Must Fit in Video Memory" Requirement

If you compress each parameter to 4 bits (that's quantization), 8 billion parameters require only about 4GB of storage space, plus a small amount of overhead and conversation cache, so the overall footprint drops significantly. This is why quantized models have become feasible on consumer-grade GPUs.

But the key question isn't "can it fit"—it's "where does it fit." Quantization only reduces the model's absolute space requirements; it doesn't change the physical constraints of running a model.

KV cache (Key-Value Cache) is the intermediate state that Transformer-based models save when generating text. Autoregressive generation outputs tokens one by one; to avoid recomputing the attention over all previous tokens at each step, the model caches the key (Key) and value (Value) vectors of already-processed tokens. The cache size is proportional to context length, number of layers, number of attention heads, and batch size—the longer the context, the larger the footprint. In long conversations or multi-turn sessions, the KV cache can reach several GB, which is one of the important reasons why an 8B model's actual video memory usage exceeds its parameter size. Some inference frameworks offer KV cache quantization or paged management to mitigate this overhead.

The Real Bottleneck: Memory the GPU Can Directly Access

When a model runs, all parameters must reside in high-speed memory that the GPU can directly address. It cannot be placed on an SSD, cannot be placed on a mechanical hard drive, and cannot even be placed solely in ordinary system memory—because the bandwidth at which the GPU accesses system memory over PCIe is far lower than video memory bandwidth, and inference speed would slow to an unacceptable level.

In other words, what you see when downloading a model is disk usage, but what you need to run a model is video memory usage. Between these two lies a hard hardware threshold. Many people mistakenly assume "if I can download it, I can run it," but in reality, video memory capacity is the first barrier that determines whether a model can run at all.

Video memory bandwidth vs. PCIe bandwidth is the key to understanding "why it can't go in system memory." Taking common consumer-grade GPUs as an example, video memory bandwidth can reach hundreds of GB/s or even over a thousand GB/s, while the theoretical bandwidth of PCIe 4.0 x16 is about 32GB/s, with actual effective bandwidth even lower. During inference, every token generated requires reading a large number of parameters; if the parameters reside in system memory, the GPU must repeatedly move data over PCIe, creating a severe bottleneck that can slow things down by more than an order of magnitude. Some frameworks support offloading some layers to system memory to run models that exceed video memory, but the cost is precisely the significant slowdown caused by this bandwidth gap.

What This Means in Practice

When choosing a model, first check the video memory requirement after quantization, then compare it against your GPU's actual available video memory (note that the system and other programs also consume some). A 4-bit quantized 8B model typically needs 5-6GB of video memory to run smoothly, while the 16-bit version needs close to 17GB. If your graphics card has only 8GB of video memory, the former is feasible, while the latter is basically hopeless.

Understanding the relationship between parameters, precision, quantization, and video memory can help you avoid the awkward situation of downloading a model only to find it won't run, and can help you more rationally assess what kinds of models your hardware is suited for.

One-line Summary

Model parameters are not the same as file size—video memory capacity is the real barrier to running open-source AI models.

Paragraph Summary

The article clears up a common misconception: a model labeled 8B means 8 billion parameters, not an 8GB file. The space each parameter occupies depends on storage precision—at FP16, 8 billion parameters take about 16GB, and with framework overhead and KV cache, actual video memory usage can exceed 17GB; quantizing to 4 bits can compress the parameters to about 4GB. But quantization only lowers space requirements; it doesn't change the physical constraint that running a model requires it to reside in video memory the GPU can directly address—the model cannot be placed on a hard drive or in ordinary system memory, or the PCIe bandwidth bottleneck will make inference unacceptably slow. Therefore, when choosing a model, first check the video memory requirement after quantization, then compare it against the GPU's actual available video memory: a 4-bit quantized 8B model typically needs 5-6GB, feasible on an 8GB graphics card, while the 16-bit version is basically hopeless.