Running a 104GB Large Model on 48GB RAM: A New Breakthrough in Local Inference on Mac

A Technical Experiment That Defies Conventional Wisdom

On Hacker News, a Show HN post titled "Running the 104GB Qwen3.8-Flash-Next model on a Mac with 48GB of RAM at ~12 tok/s" sparked heated discussion, quickly garnering 126 upvotes and 81 comments. At first glance, the numbers seem almost counterintuitive — how can a model weighing in at 104GB possibly run on a Mac with only 48GB of unified memory?

Behind this lies a technical direction in local large language model (LLM) inference that deserves deep exploration: how to break through the physical limits of hardware memory and enable consumer-grade devices to handle ultra-large-scale models. For developers and tech enthusiasts interested in local AI deployment, this is an exceptionally valuable case study.

How to Run an Oversized Model When Memory Falls Short

Core Principle: Memory Mapping and On-Demand Loading

To understand how 48GB of memory can run a 104GB model, we first need to break free from the assumption that "the entire model must fit in memory." Modern inference frameworks widely employ memory mapping (mmap) technology, which maps model weight files into virtual address space rather than loading everything into physical memory at once.

Memory mapping is a low-level mechanism provided by the operating system that maps file contents directly into a process's virtual address space. Its core concept stems from virtual memory management: the OS maintains a page table that maps virtual addresses to physical memory pages or disk locations. When a program accesses a page that hasn't been loaded yet, it triggers a "page fault," and the OS immediately reads the corresponding data from disk into physical memory. This lazy loading strategy means that even when mapping a file far larger than physical memory, the system only allocates physical memory when data is actually accessed. In LLM inference scenarios, mainstream frameworks like llama.cpp leverage mmap to map GGUF-format model weight files into memory, letting the OS page scheduler automatically manage which weights stay in memory and which get swapped to disk — achieving ultra-large model loading that's nearly transparent to the application layer.

During actual inference, the OS dynamically loads weight pages that are genuinely needed into memory based on access patterns, while pages no longer in use get swapped out to disk. This means the physical memory actually occupied at any given moment is far less than the model's total size. Combined with macOS's deep optimizations for Unified Memory Architecture (UMA) and high-speed SSD swapping, Apple Silicon devices have a unique advantage in these scenarios.

In traditional PC architectures, CPU and GPU have their own separate memory pools, and data transfer between them requires copying through the PCIe bus, which adds latency and limits available GPU VRAM. Apple Silicon's Unified Memory Architecture fundamentally changes this paradigm: CPU, GPU, and Neural Engine share the same physical memory pool, accessing it directly through on-chip interconnects without data copying. High-end chips like M2 Max/M3 Max achieve unified memory bandwidth of up to 400GB/s, far exceeding typical PC system memory bandwidth. More importantly, macOS's virtual memory subsystem is deeply optimized for SSD swapping, and paired with Apple's custom NVMe controller (with sequential read speeds up to 7.4GB/s), weight page swap-in and swap-out between disk and memory is extremely efficient. This is the hardware foundation that enables Mac devices to maintain usable inference speeds even under memory-constrained conditions.

MoE Architecture: A Natural Fit

Notably, the Qwen3 series employs a Mixture of Experts (MoE) architecture. While such models have massive total parameter counts, only a small subset of "expert" networks is activated during each forward inference pass. This sparse activation characteristic naturally complements the on-demand memory loading strategy — the weights of inactive experts can remain on disk, only being loaded into memory when selected by the router.

The core design philosophy of MoE architecture is "conditional computation" — not every inference pass uses all of the model's parameters. Specifically, MoE models set up multiple parallel "expert" sub-networks (typically 8, 16, or more) within the Transformer's feed-forward network (FFN) layers, along with a lightweight "gating network" (router). Each input token, after passing through the gating network, is routed to only the Top-K experts (usually K=2) for computation, while the remaining experts are completely inactive. Take Qwen3-235B as an example: its total parameter count is approximately 235 billion, but only about 22 billion parameters are activated per inference pass — less than 10% of the total. This design allows the model to maintain large-capacity knowledge storage while keeping computational overhead comparable to a much smaller dense model. Google's Switch Transformer and Mixtral 8x7B are classic representatives of MoE architecture, and in recent years, Chinese models like Qwen3 and DeepSeek-V3 have also adopted this architecture at scale.

It's precisely this characteristic of MoE architecture that allows a model with "104GB total size but small per-inference activation" to maintain usable inference speeds within limited memory, achieving approximately 12 tok/s. For non-real-time scenarios like reading, code assistance, and similar tasks, this speed is already practically useful.

Is 12 tok/s Fast Enough?

Balancing Speed and User Experience

A generation speed of 12 tokens per second roughly matches the pace of normal human reading. In LLM inference, "tok/s" (tokens generated per second) is the core metric for measuring generation speed. One English word typically corresponds to 1-2 tokens, while one Chinese character is roughly 1.5-2 tokens. Normal human reading speed is about 200-300 English words per minute, which translates to approximately 5-10 tokens per second. Therefore, 12 tok/s is slightly faster than human reading speed, and in streaming output mode where text appears character by character, users can read as they wait, making the experience fairly smooth. For reference, cloud-based APIs like ChatGPT typically generate at 30-80 tok/s, while running small models locally (such as 7B parameters) on Apple Silicon can achieve 40-60 tok/s. While 12 tok/s doesn't match cloud services, considering the model size (104GB) and hardware constraints (48GB memory), this performance is quite impressive.

For interactive conversations, this speed may feel somewhat sluggish; but for batch processing tasks, long-form text generation, and code completion, it's well within acceptable range.

More critically, this number is achieved under the extreme condition of "model size being more than double the available memory." Without memory mapping and MoE sparse activation, this configuration couldn't even load the model, let alone achieve any inference speed. Going from "completely unable to run" to "stable output at 12 tok/s" is itself a qualitative leap.

Disk I/O Becomes the New Performance Bottleneck

When model weights need to be frequently swapped between disk and memory, SSD read speed becomes the key factor determining inference performance. This also explains why Apple Silicon Macs excel at these tasks — their high-bandwidth NVMe storage and unified memory architecture dramatically reduce the latency of weight swapping.

It's foreseeable that as faster storage solutions like PCIe 5.0 SSDs become more widespread, the performance of this "trading disk for memory" inference approach will continue to improve. PCIe (Peripheral Component Interconnect Express) is the bus standard connecting CPUs to high-speed peripherals, with bandwidth doubling each generation: PCIe 3.0 x4 lanes provide approximately 3.5GB/s bandwidth, PCIe 4.0 x4 about 7GB/s, and PCIe 5.0 x4 can reach 14GB/s. Current consumer-grade PCIe 4.0 NVMe SSDs commonly achieve 5-7GB/s sequential read speeds, while PCIe 5.0 SSDs now entering the market can reach 10-14GB/s. Additionally, new interconnect protocols like CXL (Compute Express Link) allow memory pool expansion through high-speed buses, potentially offering more flexible memory-storage tiering solutions for local LLM inference in the future. This means the "trading disk for memory" inference strategy still has considerable room for performance improvement with future hardware generational upgrades.

Key Takeaways for Local LLM Deployment

The Capability Boundary of Consumer Hardware Is Expanding

The greatest significance of this experiment is that it redefines the capability boundary of consumer devices for running large models. Previously, it was widely believed that running models in the hundred-GB range required high-end GPU clusters in data centers. This case proves that through clever software design, an ordinary developer's Mac can handle models that were once considered out of reach.

This directly benefits a variety of real-world needs: keeping data local in privacy-sensitive scenarios, deployment in offline environments, and reducing API call costs. It moves local AI from the stage of "only able to run small models" toward a new phase of "also able to run large models."

Dual Engines: Quantization and Architectural Innovation

It's worth noting that breakthroughs like this are typically the result of multiple technologies stacking together: model quantization (such as 4-bit or 2-bit compression) reduces weight file sizes, MoE architecture lowers the activation cost per inference pass, memory mapping solves the loading bottleneck, and high-bandwidth storage on the hardware side provides performance guarantees. No single technology could independently achieve these results.

Model quantization is a technique that compresses neural network weights from high-precision floating-point numbers (such as 16-bit FP16 or 32-bit FP32) to low-precision integer representations. Common quantization levels include INT8 (8-bit), INT4 (4-bit), and even aggressive 2-bit quantization. Take an original FP16-format 70B parameter model as an example: its weight file would be approximately 140GB; after 4-bit quantization, the size can be compressed to about 40GB — a roughly 3.5x reduction. The core challenge of quantization lies in controlling precision loss — mainstream quantization schemes like GPTQ, AWQ, and GGUF use strategies such as group quantization and non-uniform quantization to dramatically compress size while keeping model performance degradation within acceptable bounds. In this case, the 104GB model file has very likely already undergone some degree of quantization (the original FP16 format would be much larger), and it's the combination of quantization, mmap, and MoE that makes running on a 48GB memory device possible.

This also demonstrates that progress in local LLM deployment won't come from hardware alone — it's a systems engineering effort requiring hardware-software co-optimization. In the comment section, numerous developers were already discussing specific approaches to further reduce memory footprint and improve inference throughput, reflecting the tech community's intense interest in this direction.

Conclusion

From this case of running a 104GB model on 48GB of memory, we see not just a stress test, but a substantive expansion of what's possible in local AI deployment. It tells us that physical hardware limitations are not insurmountable — through the organic combination of memory mapping, MoE sparse activation, model quantization, and other techniques, consumer-grade devices are fully capable of handling ultra-large-scale models.

For developers looking to explore large models locally, experiments like this provide a clear and actionable reference path. As these technologies continue to mature and hardware capabilities keep improving, the era when everyone can run cutting-edge large models on their personal devices is accelerating toward us.