Running a 125B Model on a Single RTX 5090: Strata Benchmarks Qwen3.8 Flash Next

Consumer GPU Runs a Data-Center-Scale Model — A Common Assumption Gets Broken

"32GB of VRAM can't fit a hundred-billion-parameter model" — this has been treated as an iron law in local LLM deployment. By conventional wisdom, models like Qwen3.8 Flash Next, with its 125 billion parameters, belong exclusively to data centers and cloud APIs, well out of reach for an ordinary home PC.

An open-source framework called Strata is rewriting that rule. According to a hands-on demo by a Bilibili content creator, his home gaming rig — a single RTX 5090 paired with 64GB of RAM — successfully ran the 125B Qwen3.8 Flash Next, achieving a decode speed of 113 tokens per second in short-context conversations. That's a remarkably strong result for local inference.

The core tension is real: 32GB of VRAM physically cannot hold hundred-billion-scale weights. Strata doesn't violate that physical constraint — instead, it uses a layered scheduling mechanism that makes the model appear to run fully on consumer hardware.

Three Layers: Fitting a Hundred-Billion-Parameter Model into a Home PC

The secret behind this feat unfolds in three layers, each addressing the VRAM shortage in a different way.

Layer 1: On-Demand MoE Expert Scheduling

The first layer is dynamic scheduling of MoE (Mixture of Experts) architecture. The model's many experts don't all need to reside in VRAM at once — hot experts stay resident in VRAM, while the rest "sleep" in system RAM and on SSD, paged in only when needed. This is precisely why MoE architecture is naturally suited for local deployment: each inference step activates only a small fraction of the total parameters.

The monitoring dashboard shows over 11,000 experts in the resident cache, with a cache hit rate of 94.9%. A high hit rate means most inference requests can be fulfilled quickly within VRAM, without frequently fetching from SSD — and that's the key to maintaining speed.

MoE (Mixture of Experts) has become the dominant approach for scaling up large models in recent years. Unlike traditional dense models — where every token passes through all parameters — MoE replaces FFN layers with multiple parallel "expert" sub-networks. A lightweight gating network (Router) dynamically selects the Top-K experts for each token, while all other experts sit out entirely. This means the model's active parameter count is far smaller than its total parameter count. Qwen3.8 Flash Next may have 125 billion total parameters, but the number actually activated per token could be just tens of billions — making its compute cost comparable to a dense model of similar activation size. Strata exploits this property: expert weights not selected in a given inference step can safely leave VRAM and be loaded back from RAM or SSD when the router calls for them. The expert hit rate (94.9%) directly determines the real-world latency of this mechanism: the higher the hit rate, the fewer cross-device weight transfers are needed, and the closer performance gets to pure VRAM inference.

Layer 2: Tiered KV Cache Management

The second layer is tiered management of the KV cache. Only 32K units of KV cache are kept in VRAM; the rest is offloaded to system RAM. In practice, the KV VRAM hit rate reached 98.9%, meaning nearly all key-value lookups are resolved within VRAM — avoiding the expensive cost of cross-device data movement.

The KV cache (Key-Value Cache) is a foundational acceleration mechanism for Transformer inference. During autoregressive decoding, each Attention layer needs the Key and Value vectors for every previous token; recomputing them at each step would make time complexity grow quadratically with context length. KV caching stores already-computed Key/Value tensors so subsequent steps can reuse them directly, reducing time complexity to linear. The tradeoff is that VRAM usage grows linearly with context length — for a 100B+ model with long contexts (e.g., 128K tokens), the KV cache alone can consume tens of gigabytes. Strata's tiered KV cache strategy keeps hot context in VRAM (32K units) and offloads less-frequently-accessed historical KVs to RAM, trading a 98.9% hit rate for a massive expansion in context capacity. The design philosophy mirrors CPU multi-level caching (L1/L2/L3/RAM) — exploit access locality by keeping the most frequently used data in the fastest storage tier.

Layer 3: MTP Speculative Decoding

The third layer is MTP (Multi-Token Prediction) speculative decoding, which attempts to guess multiple tokens in a single inference pass. When the guesses are correct, they're accepted outright — spreading the generation cost across multiple tokens at once. This is one of the most mainstream techniques in recent LLM inference acceleration.

Speculative decoding is a family of methods that replaces sequential token-by-token generation with a "guess-and-verify" approach. In the standard implementation, a lightweight draft model quickly generates several candidate tokens, which the main model then verifies in parallel. Accepted tokens are kept; at the first mismatch, the sequence is truncated and regeneration begins, while the parallelism of the verification step dramatically improves GPU utilization. MTP (Multi-Token Prediction) is a variant: additional prediction heads are added during training or inference, allowing a single forward pass to predict multiple token positions simultaneously — achieving similar acceleration without a separate draft model. Models like DeepSeek^Ad-V3 have adopted MTP as a native training objective, making it naturally compatible with speculative verification. In Strata's context, MTP is especially valuable because: given the fixed latency cost of cross-device data transfers, each forward pass carries a relatively constant overhead. If multiple tokens can be accepted per pass, the effective token throughput multiplies accordingly — pushing performance as close as possible to the speed ceiling in a bandwidth-constrained consumer environment.

Benchmark Results: Speed and VRAM Utilization

With all three layers combined, the measured results are as follows: median decode speed for short conversations is 113 tokens/second; decode speed remains meaningful even as context grows to 128K tokens; and prefill speed at 32K context reaches 3,860 tokens/second.

The creator noted that compared to the official 5070 baseline, decode speed is roughly 2× and prefill is roughly 2.5×. He was also candid: "The methodologies differ — this is only a rough order-of-magnitude comparison." That's a commendably rigorous stance. Official benchmarks and local tests differ in test conditions, hardware, and quantization settings, so these multiples convey only a rough sense of scale and should not be taken as precise performance conclusions.

On the VRAM side, Task Manager showed 31.2 GB / 31.5 GB utilized — nearly the full 32GB. In other words, Strata squeezes every last bit of VRAM out of the RTX 5090, with virtually no headroom to spare.

Deployment Requirements and Accessibility

For users who want to try it, the barrier is lower than expected:

• 64GB RAM: can run the full-size model
• 48GB VRAM: use the Q2-quantized Coder variant
• 32GB VRAM: also works, with the appropriate quantization config

The installation process has been streamlined to a single-click open-source setup — double-click "Start Here," download roughly 70GB of model files, and you're ready. The interface layer supports both OpenAI and Anthropic API formats, meaning the vast majority of existing toolchains and applications can plug in seamlessly without rewriting any calling code.

One particularly compelling detail: the creator mentioned that the script, charts, and animations for this video were all produced by Flash Next running locally on the same machine. That's a practical demonstration of real-world utility — not just a benchmark showpiece, but a system capable of handling actual production workloads.

Why Local Large Models Matter

The value of frameworks like Strata lies in bringing hundred-billion-parameter models — previously confined to the cloud — down to consumer hardware. The practical benefits are real: data stays private (no cloud uploads required), there are no API call costs, and the system works fully offline.

That said, a grounded perspective is warranted. A 94.9% expert hit rate still means roughly 5% of requests must reach slower RAM or SSD. In scenarios with long, complex contexts or where niche experts are frequently called upon, speeds may drop noticeably. The data in this article comes from a single creator's live demonstration; broader validation across diverse hardware configurations is still needed.

Nonetheless, the assumption that "consumer GPUs can't run hundred-billion-parameter models" has already been pried open — by a single RTX 5090 and a clever layered scheduling system. The project is open source on GitHub; interested readers can search for Strata and replicate the experiment themselves.