Strata Local Benchmark: Running a 180B MoE Model on a 12GB GPU

Why Can a 180B MoE Model Run on a Consumer GPU?

A 180B-class sparse Mixture-of-Experts (MoE) model used to mean two H100s or hourly cloud rental fees. A recent hands-on test by a Bilibili creator offers a completely different path: using a local inference engine called Strata to run this kind of behemoth on a single consumer GPU with just 12GB of VRAM.

The model under test is Qwen3.8-Flash-Next — a 48-layer architecture with 512 experts per layer and 180B-class parameters (125B main body + 51B N-gram table + 4B MTP). On a machine with 12GB VRAM and 64GB RAM, official figures show: short conversations reach 93 tokens/sec, while reading a 32K-token prompt hits 2,170 tokens/sec.

For anyone used to the idea that "large model = large VRAM," these numbers are counterintuitive. The key is that it doesn't cram the entire model into VRAM — instead, it uses a tiered scheduling approach that places the right data on the right storage medium.

Three Prerequisites: Sparsity, Quantization, and Tiered Caching

Three mutually reinforcing mechanisms make it possible to compress VRAM usage this dramatically.

First: sparsity. The essence of MoE is conditional computation — each token only activates roughly 10 out of 512 experts. The vast majority of parameters are "dormant" during any single inference pass and don't need to reside in fast VRAM.

Second: quantization. The approach uses a GSQ-RCO mixed-precision quantization strategy that assigns bit-widths per tensor. The IQ3S tier averages about 3.5 bits, with quality reportedly close to BF16. Quantization compresses the model to less than a quarter of its original size — the physical prerequisite for fitting it into consumer memory.

Third — and most critically — experts are cached across three tiers. The hottest experts go into VRAM, the next warmest into RAM, and cold experts are read directly from SSD, with the router predicting and prefetching them one layer ahead. The 28.8GB N-gram interpolation table stays on disk entirely, consuming neither VRAM nor RAM. This "hot-cold separation" ensures that limited VRAM only serves the truly high-frequency computation paths.

The sparse activation principle in MoE is worth explaining further. Traditional dense models activate all parameters on every inference pass, whereas MoE includes a large number of "expert" sub-networks in each Transformer layer (512 in this case), with a lightweight "router" network deciding which experts handle each token. Since only a tiny fraction of experts (~10) are activated at a time, the model's total parameter count (180B) far exceeds the parameters actually involved in any single inference (effective activation is roughly 20–30B). This is the core advantage of sparsity. Router prediction (prefetching) goes a step further: by analyzing routing decisions from the previous layer to anticipate which experts will be activated next, it moves them from RAM or SSD into VRAM ahead of time, parallelizing data transfer with computation and hiding the latency of cross-medium access.

Test 1: Ideal DECODE Speed in Short-Context Scenarios

The test platform is fairly powerful: an RTX 4080-class 32GB + RTX 5070Ti 16GB dual-GPU setup totaling 48GB VRAM, 128GB quad-channel DDR4, and a 32-core CPU. Strata includes a built-in web interface with Chat, Monitor, and About tabs for real-time load monitoring.

The first test used a very short prompt and let the model generate continuously. The monitor panel showed GPU load, VRAM usage, PCIe bandwidth, CPU usage, and disk reads all ticking in real time. The model ultimately produced 11,665 tokens in 130.8 seconds at an average of 90.4 tokens/sec.

The expert cache hit rate in this round was 98.0% — nearly everything hit VRAM. With such a short context, this represents the model's optimal decoding speed.

Test 2: 100K-Token Long Context — Speed Drops Only 4%

The real stress test for the scheduling system is long context. The second test pushed the context to the 100K-token range: 101,713 tokens as input, with 101,333 freshly read in at 2,964 tokens/sec, with an instantaneous peak of 3,049. This used roughly 40% of the 256K window.

Immediately after reading, the model generated 1,849 tokens at 86.8 tokens/sec. In other words, even as context ballooned from a few hundred to 100K tokens, decoding speed dropped by only about 4%. This demonstrates that the tiered caching and prefetch mechanism maintains high hit rates even under long sequences, with no performance cliff caused by VRAM being squeezed out.

Throughout the test, VRAM held steady at 45.4GB (29.9GB and 15.6GB across the two GPUs), system RAM used only 79.6GB with 48GB still free. During prompt reading, both GPUs drew 352W — well below the 620W ceiling. Across both requests, 101,718 tokens were read and 13,514 were written, entirely offline with no token ever leaving the local machine.

With This Level of Compression, Does Quality Hold Up?

Aggressive quantization always raises concerns about precision loss. According to the vendor's benchmark data, the IQ3S tier scores 92.93 on GPQA Diamond (BF16 baseline: 91.92) and 86.86 on LiveCodeBench V6 (vs. 87.43), with an average of 93.26 vs. 93.12 across three tasks — essentially on par. And the model weighs less than a quarter of the BF16 original.

It's worth noting: these figures come from vendor benchmarks, not local measurements — that distinction should be kept in mind. From the creator's hands-on experience, the quality-to-speed tradeoff lands within a usable range.

More VRAM means faster speeds. The vendor estimates a 24GB GPU can reach approximately 100–140 tokens/sec; the dual-GPU result of 105.8 falls right in that range.

GSQ-RCO mixed quantization is the key technique for preserving precision in this setup. Traditional uniform quantization treats all weights equally, while mixed quantization dynamically assigns bit-widths based on each tensor's sensitivity to model output: tensors with greater impact on accuracy retain higher bit-widths, while less critical expert weights are compressed to lower bit-widths. The "I" in IQ3S stands for Importance-aware, "Q3" means approximately 3-bit quantization, and "S" is the Small variant, averaging about 3.5 bits. By comparison, BF16 uses 16 bits per parameter and INT8 uses 8 bits — so 3.5-bit quantization compresses the model to roughly 22% of its original size. GPQA Diamond (graduate-level science Q&A) and LiveCodeBench (competitive coding problems) are both mainstream benchmarks for evaluating frontier model reasoning, and near-parity with BF16 on both suggests that quantization loss is concentrated at extreme edge cases, with limited impact on everyday use.

Getting Started: One-Click Launch and Tier Selection

The barrier to entry is kept low. Running Starter.bat once automatically installs the engine and all dependencies — no compiler or CUDA Toolkit required. The recommended configuration is 12GB VRAM + 64GB RAM or above.

Tiers are selected based on available VRAM and RAM:

• Q20 at 66GB: fastest
• IQ2XS at 68GB: more balanced
• IQ3XS at 76GB: best quality

The IQ3S used in this test requires 83.6GB. The engine includes a built-in Calibrate function that determines the fastest settings for your machine in about 10 minutes. There's also an experimental option: the 111GB Answer Q4 model, which can run within a 40GB memory budget at around 7–8.5 tokens/sec.

An Engineering Perspective: Bringing the Data Center to Your Desk

The most significant thing about this approach isn't any single benchmark number — it's how the combination of sparse activation, mixed quantization, and hot-cold tiering redefines the boundary of "what models can run locally." A 180B-class MoE model has moved from the data center back to the desktop, running fully offline with no data leaving the machine — a clear value proposition for privacy-sensitive use cases and individual developers.

Of course, mechanisms like SSD reads and router prefetching place implicit demands on disk speed and memory bandwidth, and real-world performance will vary by machine. The Calibrate function is your most reliable guide. But the direction is clear: models keep getting larger, and the engineering innovation bringing them back to personal hardware is keeping pace.

The N-gram interpolation table (the 51GB component mentioned above) plays the role of a "statistical prior" in LLM inference: it stores probability distributions for vast numbers of high-frequency word sequences, allowing direct table lookups during autoregressive decoding to skip certain neural network computations, significantly accelerating the Prefill phase (i.e., reading in the prompt). The remarkable 2,964 tokens/sec read speed in this test is partly attributable to this. Because the table is large (51B parameters) and its access pattern skews toward sequential reads, keeping it on SSD rather than RAM is a sensible engineering tradeoff — SSD sequential read bandwidth is more than sufficient for prefetch needs, while freeing up precious RAM for hot expert weights. This also illustrates that the system's careful partitioning across storage tiers (VRAM → RAM → SSD) is not a simple overflow fallback, but a targeted optimization matched to the distinct access patterns of different data types.