Running 125B MoE on a 12GB GPU? Inside Strata's Three-Tier Cooperative Engine

The Reality of Running Large Models on Consumer GPUs

Running a 125B-parameter MoE model locally on a desktop used to be synonymous with "multi-GPU server." Not enough VRAM means constant page swapping, and speed drops to an unusable level. The open-source C++ inference engine Strata offers a new approach: instead of stacking up VRAM, it organizes three storage tiers—VRAM, system memory, and NVMe SSD—into a pipeline, enabling a consumer card with 12GB to 16GB of VRAM to achieve 40 to 95 tokens/s generation speed.

Whether that number holds up depends on three things—choosing the right model, having enough memory, and understanding the tiering. Here's the breakdown in that order.

Prerequisite: The Model Architecture Must Match

Strata is designed for extremely sparse MoE models. Take Kunon 3.8 FlashNext as an example: a 125B-parameter backbone with 24,576 experts total, 512 routed experts per layer of which only 10 are activated per token, meaning the actual parameter count involved in a single computation is roughly 6B. In other words, 125B is the "warehouse capacity," and 6B is the "amount moved each time"—that's the fundamental reason small VRAM can run it.

Conversely, dense models like QN3.8 27B require reading all 27B weights in full for every token, and cannot use Strata's sparse scheduling strategy. Choose the wrong model, and all the hardware optimization downstream is moot.

MoE (Mixture of Experts) architecture is the key prerequisite for understanding this article. Unlike dense models where every token passes through all parameters, MoE sets up multiple parallel "expert" sub-networks in each layer and uses a router network to select only a few experts per token for computation. Therefore, the model's total parameter count (warehouse capacity) and the actual computation per pass (activated parameters) are two different things. The 125B backbone in this article can be driven by small VRAM precisely because—although it stores 24,576 experts—each token activates only 10, about 6B parameters.

Quantization levels (e.g., IQ2, IQ3) refer to precision tiers for low-bit compression of model weights. The IQ series is an importance-weighted quantization method; smaller numbers mean more aggressive compression, smaller footprint, but greater precision loss. IQ2 is roughly just over 2 bits per weight, IQ3 about 3 bits. This is also the key to fitting a 125B model into 48-64GB of memory and tens of GB of SSD—without quantization, the weights alone would require hundreds of GB.

Three-Tier Hardware Division: VRAM, Memory, and SSD Each Play Their Part

Beyond the 125B backbone, the model also carries an Ngram table of about 51B parameters. Strata handles this extra weight by mapping it to the NVMe SSD and reading it on demand, rather than keeping it resident in memory.

The rest is tiered by access frequency: the entire pool of 24,576 experts goes into system memory, while the attention layers and several thousand high-frequency hot experts stay in VRAM. For daily operation, IQ2 or IQ3 quantization is recommended, with 48GB to 64GB of system memory and 70GB to 80GB of NVMe SSD space.

The logic behind this tiering isn't complicated: VRAM holds the hottest data, memory holds the next-hottest expert pool, and the SSD catches cold data. The real challenge is that if you wait for memory after each single word pops out, speed gets completely dragged down by memory access latency.

The Ngram table can be thought of as a "phrase lookup table": it records common consecutive word fragments in text and their likely continuations, used during inference to quickly provide candidate words and boost hit rates in conjunction with speculative decoding. In this article, the table has about 51B parameters—already close to the backbone's single-pass activation size—so it's not suited to reside in VRAM or memory, and is instead placed on NVMe for on-demand reads. This also explains why the SSD's random read performance (rather than sequential bandwidth or capacity) directly affects first-token latency—table lookups are typical small-block random access, where IOPS and latency matter more than large-file copy speed.

Two Keys to Avoiding Stalls: Speculative Decoding and Chunked Prefill

Strata uses two methods to amortize memory access costs.

The first is speculative decoding. The model includes an MTP multi-token prediction layer that can draft multiple subsequent tokens at once, then verify them in batch, merging what would have been multiple serial memory accesses into one batched operation.

The second is chunked prefill. For long-input scenarios, prefill is split into chunks and processed step by step; the official prefill speed on a fill-in card can reach 1000 to 2600 tokens/s. Only with these two combined does 40 to 95 tokens/s generation speed become possible on consumer hardware.

Speculative Decoding is a technique that trades "small fast steps + batch verification" for speed. Conventional autoregressive generation requires a full forward pass and a weight access for every token produced; speculative decoding first uses a lightweight draft model (in this article, the model's built-in MTP multi-token prediction layer) to guess multiple candidate tokens at once, then has the main model verify these candidates in parallel in a single forward pass, accepting the correct ones. Since memory access is done in batches, the VRAM/memory access overhead per token is significantly amortized—this is one of the core mechanisms enabling high tokens/s on small VRAM.

MTP (Multi-Token Prediction) is an auxiliary training objective that has the model predict tokens at multiple future positions simultaneously. It naturally produces a prediction head that can quickly draft subsequent tokens, so during inference it can directly serve as the draft source for speculative decoding without training a separate small model.

Hardware Configuration and Launch

The hardware requirements boil down to three: a GPU with 12GB to 16GB of VRAM, 48GB to 64GB of system memory, and 70GB to 80GB of NVMe SSD space.

The launch process is fairly friendly for ordinary users. On Windows, just run starthere.bit; on Linux, run the setup.10 script. The script automatically detects hardware and matches quantization levels such as IQ2XS or IQ3IXS, then starts a local service on port 8080 with OpenAI- and Anthropic-compatible APIs.

Selection Boundaries

• If you have 48GB+ memory, 12GB+ VRAM, and want to experience a 125B-class MoE locally and hook it into a coding assistant: Strata is a suitable choice.
• If you only have 6GB or 8GB VRAM and want to run a 35B-class MoE: check out FreeToken.
• Running a 27B dense model: you need a dedicated dense inference engine; Strata's sparse scheduling won't help.

Practical Advice

The core of this solution isn't the marketing pitch of "running large models on small VRAM," but squeezing the activation characteristics of sparse MoE to the limit. In actual deployment, system memory capacity deserves priority over VRAM—48GB is the baseline, 64GB is safer. Also, placing the Ngram table on NVMe means the SSD's random read performance directly affects first-token latency; picking a drive with solid random read performance is more meaningful than simply chasing capacity. Feel free to share your configuration and measured speeds in the comments.

One-line Summary

Strata uses three-tier storage to run a 125B sparse MoE on 12-16GB consumer GPUs at 40-95 tokens/s.

Paragraph Summary

Open-source C++ inference engine Strata offers a new path to running a 125B-parameter MoE model on consumer GPUs: instead of stacking VRAM, it organizes VRAM, system memory, and NVMe SSD into a three-tier pipeline, enabling 12GB-16GB consumer cards to achieve 40-95 tokens/s generation speed. The prerequisite is that the model must be an extremely sparse MoE—the 125B backbone activates only about 6B parameters per token, and dense models like 27B can't use this approach. For storage, high-frequency experts and attention layers reside in VRAM, the 24,576-expert pool goes into system memory, and the ~51B-parameter Ngram table is mapped to the SSD for on-demand reads. Combined with speculative decoding and chunked prefill to amortize memory access costs, plus the recommended 48-64GB memory and 70-80GB SSD space, you can use the launch script to auto-match quantization levels and start a local service.