Demystifying Unified Memory Architecture: How Mini PCs Run 70B Large Models

A Counterintuitive Phenomenon

When it comes to running large language models locally, most people's first instinct is: the more powerful and expensive the GPU, the larger the model it can run. Yet in reality, there's a surprising phenomenon—a mini PC equipped with a unified memory architecture can smoothly run a 70-billion-parameter (70B) large model, while an expensive discrete GPU falls short.

The key behind this isn't a contest of raw compute power, but a fundamental difference in memory architecture. Understanding this is crucial for any developer or AI enthusiast hoping to deploy large models locally.

The "VRAM Wall" Dilemma of Discrete GPUs

VRAM Capacity Is a Hard Ceiling

Discrete GPUs (such as NVIDIA's consumer-grade products) have dedicated video memory (VRAM) that is physically isolated from system memory. This architectural design traces back to the GPU's origins as a specialized graphics accelerator—early GPUs were used solely for rasterization rendering, and dedicated high-speed GDDR VRAM was designed to meet the extremely high bandwidth demands of texture sampling and frame buffering. However, as GPUs expanded into general-purpose computing (GPGPU) and AI inference scenarios, the limitations of physical isolation gradually surfaced: data must be explicitly transferred between host memory and device VRAM over the PCIe bus, which not only introduces latency but is also constrained by the PCIe bandwidth ceiling (PCIe 4.0 x16 offers about 64GB/s, far below the 1TB/s VRAM bandwidth of GDDR6X). This architectural contradiction has been amplified to the extreme in the era of large models.

When running large models, the model's weight parameters must be fully loaded into VRAM to enable efficient inference.

Taking a 70B-parameter model as an example, understanding why so much space is needed requires grasping a basic conversion rule: each parameter occupies 2 bytes in FP16 (half-precision floating point) format, so 70 billion parameters at full precision require about 140GB of memory. Quantization techniques compress the size by reducing the numerical precision of each parameter—4-bit quantization compresses each parameter to 0.5 bytes, theoretically shrinking 140GB down to about 35-40GB (the actual figure varies slightly depending on the quantization scheme and additional overhead). Even so, even with 4-bit quantization, more than 40GB of VRAM is still required. Yet mainstream consumer GPUs typically have between 8GB and 24GB of VRAM—even the flagship RTX 4090 has only 24GB. This means that no matter how powerful the compute cores are, the entire 70B model simply cannot fit into VRAM.

The underlying principles of quantization are worth exploring in depth. Quantization is essentially a numerical precision compression technique, with a history dating back to the field of digital signal processing. In large language models, quantization maps weight parameters originally stored in FP32 (32-bit floating point) or FP16 (16-bit floating point) to lower-bit-width integer representations (such as INT8, INT4, or even INT2), thereby dramatically compressing the model size. This process is similar to converting a high-definition image into a lossy compression format—the file becomes smaller, but some detail is lost. Modern quantization schemes (such as the K-quant series in the GGUF format) introduce a "mixed-precision" strategy: they preserve higher precision for more sensitive layers in the model (such as the key matrices of the attention mechanism), while aggressively compressing less impactful layers, achieving a better overall quality-to-size balance.

The precision loss introduced by quantization is typically measured by the "Perplexity" (PPL) metric—which reflects the model's prediction uncertainty on test text, with lower values indicating higher quality. Going from FP16 to Q4_K_M, the PPL increase for mainstream 70B models typically falls in the 0.1 to 0.3 range, which is imperceptible to the naked eye in actual conversation and reasoning tasks. Community practice suggests: if hardware capacity allows, prioritize Q5 or Q6 quantization; if capacity is tight, Q4_K_M is widely regarded as the "minimum acceptable baseline"; Q3 and below cause noticeable degradation in output quality and are generally not recommended for production scenarios. Common schemes include variants like Q4_K_M and Q5_K_S in the GGUF format, as well as frameworks like GPTQ and AWQ, offering ordinary users flexible choices.

The Cost of VRAM Overflow

When a model exceeds VRAM capacity, the system is forced to repeatedly shuffle data between VRAM and system memory. This "memory swapping" causes catastrophic performance degradation, slowing inference speed to the point of being practically unusable. Faced with the dilemma of insufficient single-card VRAM, the industry has also developed multi-GPU VRAM aggregation solutions: combining the VRAM of multiple cards via NVLink, or splitting model layers across multiple cards through tensor parallelism. However, these solutions are almost impractical for consumer users—NVLink support has been largely discontinued in consumer product lines (retained only in data center cards like the A100/H100), multi-GPU parallelism requires complex software configuration, and each additional RTX 4090 costs over 10,000 RMB. For large models, VRAM capacity matters far more than compute speed—if it doesn't fit, nothing else matters.

How Unified Memory Architecture Breaks Through the Bottleneck

What Is Unified Memory

Unified memory architecture breaks the physical isolation between CPU memory and GPU VRAM. The CPU and GPU share the same physical memory pool, eliminating the need to copy data between them. Apple's M-series chips are the most representative consumer product of this architecture—Apple launched the M1 chip in 2020, integrating compute units such as the CPU, GPU, and Neural Engine into a single SoC (System on Chip). The M1 Ultra supports up to 192GB of unified memory, and the M2/M3 series have continued and strengthened this design.

The technical evolution of SoC architecture and unified memory has a profound engineering background. An SoC (System on Chip) integrates traditionally separate components—the processor, graphics unit, memory controller, Neural Engine, and more—into a single chip, a design that first gained widespread adoption in mobile devices (smartphones, tablets). Apple brought this approach to the desktop and laptop markets. Its unified memory is not "shared memory" in the ordinary sense (which usually refers to integrated graphics carving out a portion of system memory), but rather physically packages memory chips directly onto the SoC substrate, sharing the same memory bus with the CPU and GPU, eliminating the latency and bandwidth bottlenecks of PCIe bus transfers. This gives Apple's M-series unified memory bandwidth (up to 800GB/s on the M3 Ultra) that far exceeds traditional x86 platform APU solutions, and it is the technical root of Apple platforms' unique competitiveness in local AI inference scenarios.

Beyond Apple, AMD's APU (Accelerated Processing Unit) series similarly integrates the CPU and Radeon GPU to share system memory, and mini PCs equipped with Intel Arc integrated graphics adopt a similar shared-memory model, further broadening the application of this architecture on x86 platforms. This means that if a mini PC is equipped with 64GB or even 128GB of unified memory, the GPU portion can directly access the entire memory capacity. The 40GB+ of space required to run a 70B model can be easily handled by such configurations.

It's worth noting that in actual deployment, memory requirements are not limited to static weights alone. During inference, large language models maintain a KV Cache (key-value cache) to store the key (Key) and value (Value) tensors already computed in the attention mechanism, avoiding redundant computation for historical tokens. The size of the KV Cache grows linearly with context length: for a 70B model supporting a 128K context, when processing long documents, the KV Cache itself may consume an additional tens of GB of memory. This means that "being able to fit the model weights" is only the first step—the actual usable context window length is likewise constrained by total available memory. The large-capacity nature of unified memory architecture offers a significant advantage in this dimension as well.

At the toolchain level, mainstream inference frameworks such as llama.cpp and Ollama offer deep optimization support for Apple's Metal API as well as Vulkan and ROCm backends, enabling non-Apple unified memory devices to efficiently run quantized large models as well. The rise of the llama.cpp ecosystem is itself a piece of technical history worth noting: llama.cpp was released by Georgi Gerganov in March 2023, initially drawing widespread attention for implementing CPU inference of Meta's LLaMA models in pure C/C++. Its core design philosophy is extreme portability and hardware independence—through abstracted support for different hardware backends (Apple Metal, CUDA, Vulkan, ROCm, OpenCL, etc.), it allows the same quantized model format (GGUF) to run on nearly all devices, from a Raspberry Pi to a Mac Studio. Ollama, built on top of llama.cpp, wraps a more user-friendly API and model management layer, further lowering the technical barrier to local deployment. The maturation of this ecosystem and the proliferation of unified memory architecture have mutually reinforced each other, forming the distinctive landscape of the current local AI deployment field.

The Tradeoff Between Capacity and Speed

Of course, unified memory is not without its costs. Understanding this tradeoff requires recognizing how large language model inference works: the inference process is divided into two stages—prefill and decode—which have starkly different hardware demands.

The Prefill stage needs to process all tokens in the input sequence in parallel, making it a compute-bound task. A discrete GPU with high FLOPS has a clear advantage in this stage, capable of processing thousands of input tokens in milliseconds. The Decode stage, on the other hand, generates tokens autoregressively one at a time, producing only a single token per step yet requiring reading all of the model's weight parameters—this is a classic memory-bandwidth-bound scenario, where the computation per step is minimal but the entire model weights must be read from memory into the compute units. For ordinary users' interactive conversation scenarios, the "output speed" most directly perceived by the user is precisely the decode stage's tokens/second. Therefore, bandwidth is the core metric that determines generation speed, not raw compute FLOPS.

Taking the RTX 4090 as an example, its VRAM bandwidth is about 1TB/s, making models that run within its 24GB VRAM extremely fast; unified memory bandwidth, by contrast, typically ranges from 100-800GB/s (Apple's M3 Ultra reaches 800GB/s, but ordinary mini PC APU solutions are usually in the 100-200GB/s range), resulting in a certain gap in token generation rate (tokens/second).

Therefore, unified memory may not necessarily surpass a high-end discrete GPU in pure compute performance. But the key is—it can fully load and successfully run the model, which is precisely what many discrete GPUs cannot do. For the large-model inference scenario, "being able to run" and "running fast" are two different levels of concern. Unified memory first solves the fundamental question of "can it run at all."

Hardware Selection Strategy for Local Large Model Deployment

Rethinking Hardware Evaluation Criteria

This phenomenon reminds us that when choosing hardware for local large model inference, we cannot simply apply the experience of gaming or graphics rendering. For large-model scenarios, a more reasonable priority order should be:

• Memory capacity first: Whether it can accommodate the complete model (including dynamic KV Cache requirements) is the primary consideration
• Architecture matters more than any single performance metric: Unified memory architecture has a structural advantage in specific scenarios
• Make good use of quantization: A reasonable quantization scheme (such as 4-bit or 8-bit) can significantly reduce memory requirements and is an important aid for ordinary users; whether to choose Q4_K_M or Q5_K_S depends on the optimal balance you seek between hardware constraints and output quality
• Pay attention to memory bandwidth: Given that it "can run," bandwidth determines the actual inference speed experience

The Local AI Deployment Advantages of Mini PCs

As the demand for local large model deployment continues to grow, mini PCs equipped with large-capacity unified memory are becoming an extremely cost-effective choice. Compared to multi-GPU workstation solutions, mini PCs are small in size, low in power consumption, and have an extremely low hardware configuration barrier, yet they can handle tasks that traditionally required workstation-grade GPUs, providing a low-barrier entry point for individual developers, researchers, and AI enthusiasts. Whether it's an Apple Mac Mini (M-series) or an x86 mini PC with an AMD APU, as long as the memory capacity is sufficient, combined with mature inference frameworks like llama.cpp and Ollama, they can all deliver a satisfying local large model experience.

Conclusion

The counterintuitive phenomenon of "a mini PC being able to run a 70B model that a big GPU cannot" reveals a profound technical principle: in the era of large models, the design philosophy of memory architecture can sometimes determine practical feasibility more than raw compute power.

For users pursuing local deployment of large models, understanding the core difference between the VRAM wall and unified memory will help you make wiser and more economical hardware decisions. Before purchasing a device, it's worth asking yourself one question: Can this memory hold the model I want to run?

Key Takeaways

• VRAM capacity is a hard ceiling: The physical VRAM limit of a discrete GPU (such as the RTX 4090's 24GB) determines the maximum model size it can run; VRAM overflow causes catastrophic performance degradation, and multi-GPU aggregation solutions are unfeasible for consumer users in both cost and complexity
• Unified memory breaks physical isolation: SoC architectures like Apple's M-series allow the CPU and GPU to share the same memory pool, and 64GB or 128GB of unified memory can easily accommodate a 4-bit quantized 70B model along with its KV Cache requirements
• Bandwidth is the true determinant of speed: The decode stage of large models is memory-bandwidth-bound rather than compute-bound, which is the fundamental reason discrete GPUs are faster once a model "can run"
• Quantization is a key lever for ordinary users: Mixed-precision quantization schemes like Q4_K_M dramatically compress size while retaining acceptable inference quality (with typically minimal PPL loss), making it a practical tool for crossing the VRAM threshold
• The software ecosystem is already mature enough: The broad support of llama.cpp and Ollama for multiple hardware backends allows the hardware advantages of unified memory devices to be fully unleashed