Running MoE Models Beyond RAM: Expert Offloading and SSD Streaming in llama.cpp

When a 100B+ parameter MoE model can't fit entirely into a 32GB RAM laptop, most people just give up. But the peculiar nature of the MoE architecture makes "running beyond RAM" possible: only a small subset of expert weights is activated per token, and the rest can stay on SSD and be loaded on demand. This article, based on llama.cpp's mmap mechanism and the --n-cpu-moe parameter, breaks down how to get large models running on limited hardware.

Why MoE Can Run Beyond RAM While Dense Models Can't

For dense models like Llama 3.1 70B, every token must pass through all weights. If the weights don't fit in memory, every token requires re-reading tens of GB from disk. Even at NVMe Gen4's 7GB/s sequential read speed, you'd only get a token every few seconds—practically useless.

The MoE architecture is completely different. Take gpt-oss-120b: ~117B total parameters, but only ~5.1B activated per token. Qwen3-30B-A3B has 30B total parameters with only ~3B activated. The router in each layer selects only a few experts (typically top-4 or top-8 out of dozens to 128 experts), and the rest are left untouched.

This leads to two key conclusions:

1. Attention layers, embedding layers, and shared layers are used by every token. They're small and should live in the fastest VRAM.
2. Expert FFN weights account for most of the volume but only a fraction is read at a time. These go in RAM; when RAM is insufficient, the OS pages them in from SSD.

More critically, unselected experts aren't all equally important. In practice, for the same kind of prompt (say, all Python code), the router repeatedly selects the same set of experts. The Linux page cache automatically keeps this "hot" set of experts in RAM while "cold" experts stay on SSD and are read on demand. Projects like Janas optimize at precisely this layer with prefetching and per-expert LRU caching, but llama.cpp's default mmap alone already gets you most of the benefit.

mmap's On-Demand Paging Mechanism: llama.cpp by default uses memory-mapped files (mmap) to load GGUF models—that is, it maps the model file into the process's virtual address space via the mmap() system call without immediately reading all its contents into physical memory. When the process accesses a tensor address, if the corresponding page isn't in physical RAM, the kernel triggers a major page fault, reads that page from SSD, and places it in the Linux page cache. The page cache is a reclaimable region of memory managed by the kernel: when free RAM is plentiful, pages that have been read in stay resident, and subsequent accesses to the same page are free; when RAM is tight, the kernel evicts old pages using an LRU-like policy. During MoE inference, attention layers and shared layers are accessed frequently and remain stably resident, while expert weights are dynamically swapped in and out based on router selection. It's precisely the page cache that makes "experts loaded on demand from SSD" the default behavior.

The Difference Between Major and Minor Faults: A major page fault means the target page isn't in physical memory and must be read from disk—costing milliseconds. A minor fault only requires establishing a mapping in the page table or page cache—costing microseconds. During the inference warm-up phase, major faults appear abundantly because model pages are being read in for the first time; after warm-up, most accesses hit the page cache, and major faults only reappear when switching to a new expert. This explains why monitoring the growth curve of maj_flt can tell you whether memory is thrashing—if it keeps climbing during generation instead of leveling off, the working set has exceeded RAM.

Estimating Memory Before Downloading a 60GB Model

Don't wait until the download finishes to discover it won't run. Use a script based on the gguf package to read GGUF metadata and tally by tensor group:

# estimate_moe.py - Estimate memory requirements of a GGUF model
import sys, re
from gguf import GGUFReader

path = sys.argv[1]
reader = GGUFReader(path)

expert_bytes, other_bytes = 0, 0
for t in reader.tensors:
    size = int(t.n_bytes)
    # In llama.cpp, expert tensors are named blk.N.ffn_up_exps.weight
    if re.search(r"ffn_(up|down|gate)_exps", t.name):
        expert_bytes += size
    else:
        other_bytes += size

gb = lambda b: b / 1024**3
print(f"Non-expert layers (put in VRAM): {gb(other_bytes):6.1f} GB")
print(f"Expert weights (RAM/SSD):        {gb(expert_bytes):6.1f} GB")
print(f"Total:                            {gb(expert_bytes + other_bytes):6.1f} GB")

Testing with a ~120B model's Q4 quantized version, the non-expert portion is only a few GB—an 8GB GPU can hold it. The expert portion is about 55-60GB. A machine with 64GB RAM can keep almost all of it in page cache; with 32GB RAM, half must be read from SSD, at which point SSD speed genuinely affects the experience.

Practical experience:

• RAM ≥ 70% of expert weights: speed is virtually identical to fully in-memory running.
• RAM between 40%-70%: runnable, speed fluctuates with the prompt. The first few seconds after switching topics will be slower.
• RAM < 40%: usually not worth it; choose a smaller quantization or a smaller model.

llama.cpp Configuration: Kicking Experts Off the GPU

llama.cpp now supports --n-cpu-moe and --override-tensor (-ot) to precisely control where each tensor is stored. The following configs are for a 12GB GPU:

**The Scope of `-ngl` and `-ot`**: `-ngl N` (`--n-gpu-layers`) specifies offloading the first N layers to the GPU; `99` means trying to put all layers on the GPU. But within each MoE layer there are both attention tensors and expert FFN tensors, and `-ngl` alone can't separate them, which is why `-ot` (`--override-tensor`, tensor override rules) is needed for finer-grained control. `-ot` accepts regex rules in the form `pattern=device`, e.g. `.ffn_.*_exps.=CPU` forces all expert tensors whose names match `ffn_*_exps` onto CPU/memory side, while the rest still go to GPU per the `-ngl` rule. `--n-cpu-moe N` is simpler syntactic sugar, equivalent to keeping the first N layers' expert tensors on CPU. The core idea behind both is the same: attention layers are small and used by every token, so keeping them in VRAM yields the greatest benefit; expert weights are large and only a small portion is used per token, so leaving them in RAM or SSD is more cost-effective.
# Build the latest version with CUDA
git clone https://github.com/ggml-org/llama.cpp && cd llama.cpp
cmake -B build -DGGML_CUDA=ON && cmake --build build -j --config Release

# Method 1: Offload all layers to GPU, but force experts to stay on CPU
./build/bin/llama-server \
  -m ~/models/gpt-oss-120b-Q4_K_M.gguf \
  -ngl 99 \
  -ot ".ffn_.*_exps.=CPU" \
  -c 16384 -t 8 --port 8080

# Method 2: More concise syntax, keep the first N layers' experts on CPU
# Adjust N until VRAM is just enough
./build/bin/llama-server -m ~/models/gpt-oss-120b-Q4_K_M.gguf \
  -ngl 99 --n-cpu-moe 30 -c 16384

A few lessons from trial and error:

• Don't add --no-mmap. This flag forces the entire model to load into RAM, and the process gets killed by the OOM killer. The default mmap lets the kernel decide which pages to keep and which to discard.
• Don't use --mlock. It locks all pages into RAM and simply fails with large models.
• -t (thread count) should be set to the number of physical cores, not logical threads. Expert matrix multiplication is memory-bandwidth-bound, and enabling hyper-threading often makes things slower.
• If there's VRAM headroom, reduce --n-cpu-moe to move some expert layers back to GPU. Each layer moved back typically adds a few percentage points of speed.

Diagnosing Whether SSD Is the Bottleneck

When tokens/s is lower than expected, the first step is checking whether the disk is continuously reading. On Linux:

**RSS and Page Cache Are Not the Same Thing**: The `rss` (Resident Set Size) reported by `ps` only counts pages mapped by the process itself that are currently resident in physical memory—though model data read in via mmap is also counted in RSS, so RSS fluctuates as accessed experts change. You can't simply conclude that "RSS much smaller than model size means thrashing"—you need to judge by combining `maj_flt` growth rate with `iostat`'s `rMB/s`. `vmtouch -v` instead counts from the file perspective how many pages of each model file are already covered by the page cache, and is the most direct indicator of whether "the working set fits." If `vmtouch` reports a resident ratio stuck around 40% for a long time while `iostat` shows sustained heavy read traffic, you can basically confirm SSD has become the bottleneck.
# Terminal 1: Monitor NVMe read throughput every second
iostat -xm 1 nvme0n1

# Terminal 2: Count llama-server's major page faults
PID=$(pgrep -f llama-server)
watch -n1 "ps -o maj_flt,rss -p $PID"

# Check how much of the model file is already in page cache
# Requires vmtouch: apt install vmtouch
vmtouch -v ~/models/gpt-oss-120b-Q4_K_M.gguf | tail -3

Interpreting the results:

• rMB/s in iostat continuously climbing (several GB/s) during generation means memory thrashing. The working set exceeds RAM—you need to reduce context, switch to a smaller quantization, or close Chrome.
• maj_flt growing rapidly during the first few dozen tokens and then stopping is normal—that's the warm-up phase.
• vmtouch shows the proportion of the file already resident. If you want to warm the cache before a demo, run vmtouch -t on the model file.

macOS and Apple Silicon are different—unified memory blurs the boundary between VRAM and RAM. In that case just focus on total memory and use sudo fs_usage -f diskio | grep llama to check whether the disk is being continuously read.

Conclusion

Running models larger than your memory isn't magic. It's only feasible for MoE, and only efficient when the expert working set fits in the page cache. Actions you can take right away:

1. When hardware is limited, prefer MoE over dense models. Running a 30B-A3B on a 16GB laptop is usually a better experience than an over-quantized 14B dense model.
2. Run the estimation script before downloading to understand the ratio of RAM to expert weights.
3. Use -ngl 99 with --n-cpu-moe (or -ot) to keep attention layers on GPU and experts on CPU. Keep the default mmap, and remove --no-mmap and --mlock.
4. If you frequently run large models, upgrade to an NVMe Gen4 SSD before upgrading your GPU. Random read speed directly affects performance when the cache misses.
5. Measure with iostat and vmtouch rather than guessing. Most slowness problems you encounter are thrashing, not CPU bottlenecks.

Projects like Janas are pushing this path further with router-based intelligent prefetching. But once you've mastered the basics of mmap and tensor placement, you can already run models on personal devices that required a server just a year ago.