RTX 3090 Takes on a 125B Model: Three-Tier Caching Delivers 5x Inference Speedup

Running a 125B Model on 24GB VRAM — What Makes It Possible

Deploying models with tens or hundreds of billions of parameters on consumer-grade GPUs has always been one of the most popular challenges among hardware enthusiasts. In this experiment, a Bilibili content creator set out to run the Flash Next model — nominally a 125B-parameter model — on a single RTX 3090 (24GB VRAM) paired with 24GB of system RAM. The result: a stable generation speed of 9–10 tokens per second, with short-text peaks hitting 18. That sounds nearly impossible, but it's built on a carefully engineered set of optimizations targeting a specific bottleneck: memory bandwidth.

To understand why this works at all, you first need to look at the model's architecture. The official "125B" refers to the model backbone, of which roughly 121B comes from the Mixture-of-Experts (MoE) layers, plus approximately 51B in PLE lookup tables and ~4B in the MTP module. The MTP was disabled for this experiment, and the full checkpoint weighs in close to 180B. The critical insight: although each layer contains 512 experts, only 10 are activated per token — meaning only about 6B worth of weights are actually involved in any given computation.

That's the essence of MoE architecture — as long as VRAM can hold the currently active weights plus runtime overhead, you can load experts on demand without stuffing the entire model into memory.

First Attempt: Why Only 1.83 Tokens per Second

The theory is elegant; the practice hit a wall immediately. The creator first ran the model using Q4 quantized weights (a custom fork of llama.cpp, open-sourced) and got a generation speed of just 1.83 tokens per second — frustratingly slow.

After what was described as "two and a half days" of debugging, the culprit was identified: Q4 weights weigh in at ~111GB, far exceeding the combined capacity of VRAM and RAM. Missing experts had to be fetched from the SSD, through RAM, and into VRAM. The bottleneck wasn't compute — it was bandwidth.

The numbers tell a compelling story: SSD read speed ~3 GB/s, RAM ~43 GB/s, PCIe ~32 GB/s, and VRAM bandwidth ~936 GB/s. A difference of over 300× between the slowest and fastest links.

The math checks out: at Q4 quantization, generating one token requires accessing 480 "layer × expert" combinations, totaling ~1.5GB of expert weights. In a worst-case cold state, just the SSD read alone takes ~0.54 seconds; add RAM reads, page fault handling, and actual computation, and you're looking at ~0.58 seconds per token — squarely in line with the measured 1.83 tokens/s. In other words, the entire inference pipeline was completely bottlenecked by storage bandwidth, with the GPU spending most of its time waiting for data.

---

What is quantization? Quantization compresses model weights from high-precision floating point (e.g., FP16/BF16, 2 bytes per parameter) to lower-precision integer representations, trading off storage space and memory bandwidth for efficiency. Q4 quantization compresses each parameter to 4 bits (0.5 bytes), roughly one-quarter the size of FP16. Q2K goes further, compressing to ~2 bits per parameter with a grouped quantization strategy (the "K" refers to block-wise quantization) that tries to preserve output quality under extreme compression. The tradeoff is precision loss — parameter values are rounded to a limited set of discrete levels, which can affect coherence and accuracy. For very large models, however, this is generally an acceptable engineering tradeoff: a Q2K-quantized 125B model often outperforms a higher-precision 7B model, because scale itself provides enough redundancy to absorb quantization error.

---

Three Optimizations: Cache, Compress, Overlap

Once the bottleneck was clear, the optimization strategy followed naturally: cache more to move less, compress experts to make them smaller, and overlap data movement with computation.

Three-Tier Caching and Storage Management

The first step was introducing a three-tier cache. The core idea is to stratify expert storage by access frequency: hot experts live in VRAM, warm experts stay in RAM, and only cold experts are read from the SSD.

The results were immediate — out of every 100 expert accesses, ~90 hit VRAM directly, ~8 were fetched from RAM, and only ~2 required an SSD read. The slowest link in the chain was pushed down to a 2% access rate. With Q4 weights, generation speed jumped from 1.83 to 8.89 tokens per second — nearly a 5× improvement. Caching eliminated the vast majority of redundant data movement.

---

Why does three-tier caching work so well? It exploits access locality. When processing a continuous conversation or long document, LLMs tend to repeatedly activate the same set of "hot" experts — because similar semantic content gets routed to similar experts. This pattern makes LRU (Least Recently Used) or frequency-based cache eviction strategies highly effective: by keeping the most recently and frequently called experts in the fastest storage tier, the vast majority of accesses never need to reach slow storage. The mechanism mirrors CPU L1/L2/L3 cache hierarchies, except here the "cache tiers" span VRAM, RAM, and SSD — with far larger bandwidth gaps between layers than in any CPU cache system. A 90% VRAM hit rate means that even though the model theoretically has 512 experts, the handful of experts actively used during real inference is much smaller than you'd expect — a concrete demonstration of how MoE routing tends to concentrate around a core set of experts for real language tasks.

---

Compressing the Experts Further

The second step was pushing quantization further. Switching to Q2K reduced individual experts from 3.1MB to 1.9MB, and the total weight from 111GB down to 75GB. Smaller experts mean the same cache capacity can hold more hot experts, improving hit rates even further.

The measured results: short-text average of 13.9 tokens/s, long-text average of 10.5 tokens/s, peak of ~18 tokens/s. A clear speed win in exchange for a drop in quantization precision.

Overlapping Data Transfer and Computation

The third step was pipelining the remaining data movement with compute. Repeatedly-used experts are batch-uploaded to the GPU, and while the current batch is being computed, the next batch is already being prefetched in the background. The idea is simple: keep both the GPU and the storage pipeline busy at the same time, replacing serial waiting with parallel execution.

---

The engineering behind this is known as "double buffering" or pipeline prefetching. GPUs expose asynchronous memory copy interfaces (e.g., CUDA Streams) that allow data transfers and kernel execution to run concurrently on separate streams without blocking each other. In the MoE inference context: while the GPU is running matrix multiplications on batch N's expert weights, the CPU or DMA controller is simultaneously moving batch N+1's expert weights from RAM into VRAM. As long as the transfer time doesn't exceed the compute time, overall throughput approaches the theoretical limit of "compute latency only." This is the same philosophy behind modern CPU out-of-order execution and superscalar pipelines — the goal is always to eliminate serial waiting and keep every hardware resource as fully utilized as possible. In bandwidth-constrained scenarios, the benefit of this optimization scales with the ratio of compute intensity to weight size per expert: the more compute-dense the operation and the smaller the weights, the more opportunity there is for useful overlap.

---

The Price of Having Only 24GB of RAM

With all three optimizations stacked, generation speed stabilizes at 9–10 tokens/s, with short-text peaks reaching 18. For a consumer-grade RTX 3090, making a 125B-scale MoE model practically usable is genuinely impressive.

But the cost is real: a 10,000-character long document still takes ~329 seconds to produce the first token. That painful time-to-first-token is the hard price of 24GB of RAM being insufficient — in a cold state, large numbers of experts must still be loaded one by one from slow storage. The creator openly acknowledged this: decode-phase generation speed is now usable, but long-text first-token latency and the 24GB RAM ceiling remain clear weaknesses. "More RAM would make all the difference."

What This Experiment Teaches Us

The value of this experiment isn't just that it "worked" — it's that it clearly demonstrates a viable engineering path for deploying large MoE models on constrained hardware. A few key takeaways worth remembering:

• MoE's sparse activation pattern makes "on-demand loading" feasible — you don't need to fit the entire model in VRAM;
• The real bottleneck in local large model deployment is usually storage bandwidth, not compute — optimizations should focus on minimizing cold data movement;
• The combination of tiered caching + quantization compression + pipelined prefetching is a general-purpose toolkit for squeezing performance out of consumer hardware.

For anyone looking to run large models on their own RTX 3090, this experiment offers a highly practical reference point. And the harsh RAM requirement is a reminder: if budget allows, adding more system RAM will often do more to reduce long-text first-token latency than upgrading the GPU itself.