llama.cpp MoE Cache Benchmark: Doubling 30B Model Decode Speed on a T4

Running large models on GPUs with insufficient VRAM usually means offloading some layers to CPU and waiting on the slowest link. Starting from build b11480, llama.cpp offers a new approach: instead of cramming entire layers of experts into VRAM, use a cache to keep the experts the model actually selects most often resident in VRAM.

We ran Qwen3-30B-A3B (18.6GB, doesn't fit) on a free 16GB Tesla T4, and decode speed jumped from the default 22.6 tokens/s to 43.2 tokens/s—nearly double. This feature was merged into the main branch in October, with multi-GPU support following shortly after.

What the MoE Cache Actually Caches

llama.cpp's approach is straightforward: pick out the layers whose experts all live in memory, group them by expert tensor type and shape, and allocate a slot pool in VRAM for each group. When the router selects an expert, a cache hit reads directly from VRAM; a miss triggers a PCIe upload. The eviction policy is LRU. Pools are shared, so a hot expert in layer 3 can displace a cold expert in layer 40.

Note that the cache only covers ffn_*_exps tensors—routers, attention, and shared experts are not included. Another key limitation is batch size: the cache only serves batches of up to 32 tokens. Larger batches (i.e., prompt processing) use the old offload path, but cached experts are copied directly to the GPU rather than re-uploaded.

Expert tensor naming and grouping logic: Internally, llama.cpp uses the ffn_*_exps naming pattern to identify expert weight tensors in MoE layers. Common concrete names include ffn_gate_exps, ffn_up_exps, and ffn_down_exps, corresponding to each expert's gate, up-projection, and down-projection. The cache mechanism scans all layers of the model, singles out layers containing these tensors, and groups them by tensor data type (e.g., Q4_K, Q6_K) and shape (number of experts × per-expert dimension). Experts within the same group share a VRAM slot pool, and slot size is strictly the byte size of one expert tensor at that type. Because grouping is based on "type + shape," a mixed quantization format like Q4_K_M splits experts into different groups—which is why Qwen3-30B-A3B is divided into two 24-layer groups (explained later). Understanding this is key to seeing why the script's computed slot count matches the startup log.

Before Configuring, Calculate How Big Your Experts Are

Guessing a --moe-cache-mib value is a recipe for pitfalls. The original article provides a pure-stdlib Python script that reads the GGUF header directly, replicating llama.cpp's slot calculation logic so you know how many slots your budget buys before loading the model.

For Qwen3-30B-A3B Q4_K_M, total expert size is 16.35 GiB, with 48 layers, 128 experts, and 8 used per token. Due to Q4_K_M mixed quantization, these experts are split into two 24-layer groups. Setting 10,000 MiB gives each group 1,834 slots, covering 59.7% of experts—exactly matching the number reported in the llama-server startup log on the T4.

There's a hard floor here: below a certain budget, llama.cpp simply gives up caching that group with only a warning; if no group meets the threshold, the server won't start at all. In testing, 50 MiB was the minimum, and 49 MiB errored out. This is worth heeding—a budget near the floor is worse than no cache at all: too few slots means rapid eviction, which actually slows things down.

GGUF header parsing and slot budget formula: GGUF is the model file format used by llama.cpp. Its header records tensor names, shapes, data types, and offsets as key-value pairs, readable without loading weights. The original script exploits this by parsing the header with pure stdlib, locating all ffn_*_exps tensors, grouping by type and shape, then back-computing each group's slot count as budget bytes ÷ per-expert bytes ÷ layers in group. The key detail is that "layers in group" participates in the division: since the slot pool is shared within a group, the budget is spread across each layer in that group—so for the same MiB budget, a group with more layers gets fewer slots per layer. The script also outputs the percentage of slots to total experts, helping you judge before loading whether your budget is near the hard floor and avoid the awkward "50 MiB works, 49 MiB errors" situation.

Two Ways to Run: Automatic Placement vs. Manual Takeover

The first is the easy route: keep --fit automatic placement, which accounts for cache VRAM usage. On the T4, --fit proactively placed 1,670 MiB fewer model layers to free up space for a 1,671 MiB cache.

The second is the faster route, and the one the original article recommends: use --cpu-moe to put all experts in memory and give all freed VRAM to the cache. The reasoning is simple: having an entire layer in VRAM only helps when the router selects some expert in that layer; the cache holds cross-layer high-frequency experts, giving higher hit rates.

But manual takeover means you have to do the VRAM math yourself. --cpu-moe is a tensor override—once set, --fit no longer moves layers, and if it doesn't fit, you get a warning and failure. Remember to leave enough space for dense weights, KV cache, and compute buffers.

Semantic differences between --fit and --cpu-moe: --fit is llama.cpp's automatic VRAM placement strategy. It probes available VRAM at startup, allocates model layers to GPU or CPU by priority, and reserves space for KV cache and compute buffers. With MoE cache enabled, it also factors cache usage into the budget, proactively placing fewer layers to free space. --cpu-moe is a tensor override switch meaning "all MoE expert tensors go to CPU." It takes precedence over --fit's automatic decisions—once set, --fit no longer moves those expert layers, and remaining VRAM goes to dense weights, KV cache, and expert cache. The fundamental difference: --fit does coarse "layer-granularity" placement—a layer is either entirely on GPU or entirely on CPU; --cpu-moe removes experts from GPU entirely, then loads them on demand at "expert granularity" via the cache, naturally yielding higher hit rates. This is why the original article recommends manual takeover.

Without Reading Logs, You Won't Know the Hit Rate

Cache size and hit rate are invisible by default. They're INFO-level logs from libllama, which llama-server maps to verbosity 4—above the default threshold of 3. Add -lv 4 to see cache size at startup and hit rate statistics after stopping the service.

T4 hit-rate logs show: small batches (≤8 tokens) hit 95.73%, while 9–32 token batches hit 59.91%. Large batches don't use slots but copy 46,881 experts from cache, saving substantial uploads.

Real Results on a T4

The comparison data from the original (Qwen3-30B-A3B Q4_K_M, single T4) is telling:

• --fit default: 22.6 tokens/s
• --fit + 10% cache: 23.6 tokens/s, 75.6% hit rate
• --cpu-moe only: 9.6 tokens/s
• --cpu-moe + 50 MiB cache: 3.8 tokens/s (cautionary tale)
• --cpu-moe + 10,000 MiB cache: 43.2 tokens/s, 95.7% hit rate

You might not have noticed: under --fit, 10% cache has a decent hit rate but limited speedup—it only covers the 20 layers left in memory, and sacrifices resident layers to free space. --cpu-moe with a large cache, by contrast, lets 95.7% of expert queries hit VRAM, truly unlocking performance. Prompt processing also improved from 150 to 382 tokens/s.

The author also measured 67.8 tokens/s on an RTX 5090 (19,000 MiB cache, 89% hit rate), showing larger caches still yield gains. But the author cautions this is single-model, single-prompt-set, single-host data—10% is just a starting point for measurement.

When the Cache Hurts More Than It Helps

Concurrency eats the cache. llama-server defaults to 4 concurrent slots; four users generating simultaneously means processing 4 tokens per step, each selecting its own experts, quadrupling slot demand and making LRU eviction more frequent. On the T4, four concurrent requests totaled 37.0 tokens/s—worse than single-request—and hit rate dropped from 95.7% to 92.3%. This feature suits single users or small teams, not busy online services.

Speculative decoding is similar. Draft verification feeds multiple tokens at once, enlarging batch size, and the draft model itself has no cache.

Multi-GPU has extra constraints. Budget is allocated by tensor-split or free VRAM. Cache enablement disables pipeline parallelism, and tensor-parallel mode refuses to start outright.

Output is no longer bit-identical. The cache runs experts in CUDA kernels instead of CPU, so results are no longer bit-identical, and greedy selection at temperature 0 may flip. In the original testing, under --cpu-moe, only 1 of 4 responses was character-for-character identical to the no-cache version; the rest diverged at 15 to 367 characters. Quality evaluation still requires your own eval set.

The ripple effect between LRU eviction and batch-size amplification: LRU (least recently used) eviction always kicks out the expert that hasn't been accessed for the longest time when the cache is full. Under single-request small batches, each token selects 8 experts, access patterns are concentrated, hot experts hit repeatedly, and eviction pressure is low. But concurrency or speculative decoding produces multiple tokens in one step, each selecting its own experts, multiplying instantaneous slot demand—forcing frequent swaps in and out of cache, and PCIe uploads actually increase. More insidious: an evicted expert may be needed again in the very next iteration, creating "thrashing," where the cache not only saves no bandwidth but adds management overhead. This explains why four concurrent requests on the T4 total less throughput than a single request, and why evaluating cache benefits requires replaying with real concurrency rather than looking only at single-stream hit rates.

Four Things to Do Before You Start

1. Run the original script on your GGUF to confirm how many slots --moe-cache-mib buys.
2. Start with -lv 4, verify the slot count in startup logs, and confirm there's no "budget too small" warning.
3. Replay with real prompts and actual concurrency, then read the ubatch <= 8 hit rate after stopping the service.
4. Compare decode speed against the same replay with --moe-cache-mib 0.

If your MoE model doesn't fit in a single GPU, upgrade to b11712 or higher, run the script, use --cpu-moe plus most of your free VRAM as cache, and enable -lv 4. On a T4, it's just one parameter and one restart for nearly double the speed.

One-line Summary

llama.cpp's new MoE cache boosts 30B model decode on a T4 from 22.6 to 43.2 tokens/s.

Paragraph Summary

Starting from build b11480, llama.cpp introduces a MoE expert cache that keeps cross-layer high-frequency selected experts in VRAM, uploading only misses via PCIe, with LRU eviction. Running an 18.6GB Qwen3-30B-A3B on a 16GB Tesla T4 with --cpu-moe and a 10,000 MiB cache, decode speed rose from the default 22.6 tokens/s to 43.2 tokens/s with a 95.7% hit rate. The cache only covers ffn_*_exps tensors and serves batches of up to 32 tokens. The article stresses that a budget near the floor is worse than no cache, and that concurrency and speculative decoding amplify batch size and eviction; output is also no longer bit-identical, making this suitable for single-user or small-team scenarios.