Strata Open-Source Runtime: Running a 125B-Parameter Qwen3 Model on a 12GB GPU

A 125-billion-parameter model running on a single RTX 5070 with just 12GB of VRAM — by conventional wisdom, that's impossible. Even compressed to 2-bit precision, the model is several times larger than the available memory. Yet an open-source inference runtime called Strata has done exactly that: it runs Qwen3.8 Flash Next on an RTX 5070 at roughly 90 tokens per second.

The secret isn't a more aggressive compression algorithm. It's a fundamentally different idea — the GPU doesn't need to hold the entire model. Strata turns your entire PC into a carefully orchestrated scheduling system, with VRAM, system RAM, and an SSD each playing a distinct role. This article breaks down the core architectural design behind this local inference approach.

The Model Itself: The Compute–Storage Gap in Sparse MoE

Before understanding Strata, you need to understand what makes Qwen3.8 Flash Next tick. It uses a sparse Mixture-of-Experts (MoE) architecture: while it stores 125 billion parameters, only about 6 billion are activated per token.

The mechanism works like this: in each layer, a router selects 10 active experts from a pool of 512, plus one shared expert that always runs. The result is a model with enormous storage requirements but very low per-step compute. That gap is precisely where Strata operates.

At the macro level, the model has 48 layers structured as 12 repetitions of "3 gated DeltaNet layers + 1 sparse injection." Each layer has 512 experts, which works out to roughly 24,500 expert instances in total. Strata tracks every single instance, and only maintains traditional attention caches for 12 layers — making long-context costs far lower than a standard Transformer.

The Mixture-of-Experts (MoE) architecture traces its roots to ensemble learning ideas from the 1990s and has recently been re-adopted by large-scale language models. The core idea is to split the feed-forward layers into multiple parallel "expert" sub-networks, with a lightweight router dynamically selecting a small number of experts for each input. This allows the model's total parameter count (capacity) to scale massively without significantly increasing the floating-point operations (FLOPs) per forward pass. Sparse MoE specifically refers to variants where only a tiny fraction of experts are activated per token — as opposed to dense MoE, where all experts contribute to every token. Qwen3.8 Flash Next selects just 10 of 512 experts per token, an activation ratio under 2%. This extreme sparsity creates a unique paradox at inference time: the GPU must keep all expert weights within reach, yet uses only a tiny fraction of them at any given moment. Traditional inference frameworks handle this by either demanding massive VRAM or accepting frequent weight swapping. Strata's design is built specifically around this paradox.

Three-Tier Memory Hierarchy: Turning Your Entire PC into a Compute Domain

Strata's central design transforms the entire PC into a three-tier memory hierarchy, with each level serving a distinct purpose:

• VRAM: Holds everything each token touches — the heterogeneous attention and DeltaNet mixer, the router, the shared expert, the output head, draft layers, and a cache of the hottest experts.
• System RAM: Pinned storage for all ~24,500 expert groups.
• SSD: Stores a 29GB N-gram lookup table, reading only a few rows at a time. The key distinction: this is a lookup, not a computation.

Conventional offloading copies missing experts from RAM to the GPU on demand, making data transfer a bottleneck on the critical path for every token. Strata takes a different approach: when an expert isn't on the GPU, the CPU computes it directly in RAM, while the GPU simultaneously processes experts already in its cache. RAM is no longer a slow "virtual VRAM" — it becomes a second compute domain.

"Offloading" refers to temporarily storing model weights that don't fit in VRAM in system memory or on disk, then transferring them to the GPU as needed. Frameworks like llama.cpp have long supported this, but the bottleneck is PCIe bandwidth: modern desktop PCIe 4.0 x16 tops out at roughly 32 GB/s, and in the generation phase every token triggers a weight transfer, so transfer latency stacks directly onto each generation step and speeds drop sharply as the offload ratio increases. Strata's approach is better understood as "heterogeneous compute" rather than "heterogeneous storage": the CPU and its integrated memory controller can already access DDR5 memory directly, typically at 50–100 GB/s or more with far lower latency than a PCIe round-trip. Having the CPU run matrix multiplications in-place in RAM and sending the results back over PCIe for the GPU to aggregate — while slower per expert than GPU execution — can actually yield higher throughput than "copy weights to GPU, then compute," especially when the two steps run in true parallel.

Hot Expert Caching: VRAM as a Working-Set Cache

Which experts get to occupy precious GPU slots? The answer lies in the fact that routing is far from random — under specific workloads, the same experts tend to be selected repeatedly.

Strata profiles usage across all ~24,500 experts, fills remaining VRAM with the hottest ones, and adjusts dynamically as the conversation evolves. Each cache hit means the CPU never needs to touch that expert. By rough estimate, every additional 1GB of VRAM accommodates around 700 more experts. VRAM effectively becomes a cache for the model's working set.

On the storage side, Strata uses low-bit quantization: the minimum configuration requires roughly 38GB of combined RAM + VRAM, while the highest-quality configuration needs about 55GB — making 64GB of system RAM an ideal target. The 51-billion-parameter N-gram table is kept on the SSD precisely because it's a lookup rather than a matrix multiplication; the OS page cache keeps frequently accessed data hot.

Prefill and Long Context: Hiding Transfers Behind Computation

Reading a long prompt and generating a response are two different problems. During prefill, thousands of tokens are available simultaneously. Strata splits the prompt into chunks of up to 8,000 tokens. While one layer's attention computation is running, the experts needed for the next layer are already being streamed over PCIe — transfers are hidden behind useful computation.

The impact is significant: on the test machine, processing a 32K-token prompt jumped from roughly 570 tokens/sec to nearly 1,300 tokens/sec.

Long contexts also put pressure on KV cache memory. When context exceeds 64K tokens, Strata keeps most of the KV cache in system RAM and retains only the hottest 32K tokens' worth in VRAM. The attention reads exactly the same values — only their location changes — and the freed VRAM flows back into the expert cache. At a full 262K context, the number of experts on the GPU more than doubles, pushing speed from roughly 51 to 63 tokens/sec.

Speculative Decoding: Using the Model's Own Draft Layers to Accelerate Generation

Even with memory issues resolved, generation itself is inherently sequential — every token requires a full pass through all 48 layers. Strata exploits the model's built-in multi-token prediction layers to address this: it drafts up to 3 tokens at once and verifies them all in a single forward pass through all 48 layers.

On average, each pass produces 2.4 to 3.2 tokens, with the large model making the final call on each — so the output is unchanged. For code-editing workloads, "prompt lookup" can also draft candidates from text already in the context, yielding an additional 6–11% improvement.

Speculative Decoding is a general autoregressive acceleration technique systematically developed by Google and others in 2022–2023. The core insight is that verifying a batch of candidate tokens costs far less than generating them one by one. The typical approach uses a tiny "draft model" to generate several candidate tokens, which a larger target model then verifies in parallel — accepting tokens that match its distribution and truncating at the first mismatch. Qwen3.8 Flash Next's design includes built-in Multi-Token Prediction (MTP) layers, effectively giving the model its own draft head and eliminating the need for a separate small model. This also avoids the acceptance-rate problems that arise from distribution mismatch between an external draft model and the target. This built-in approach is called "self-speculative decoding," and its acceptance rates are typically higher than external draft models because the draft head shares all upstream representations with the main network, giving it a more accurate preview of the next generation step. Strata's reported average of 2.4–3.2 tokens per pass translates to real-world wall-clock speeds more than 2× faster than naive serial generation, with an output distribution strictly equivalent to step-by-step sampling.

Benchmark Numbers and an Interesting Reversal

With all techniques combined, Strata's self-reported numbers on an RTX 5070 paired with a 6-core Ryzen and 64GB RAM are:

• ~90 tokens/sec for short conversations
• ~67 tokens/sec at 128K context
• ~1,300 tokens/sec during prompt prefill

There's a telling reversal worth noting: the older RTX 3090 is estimated to reach 100–140 tokens/sec. Not because it's faster — but because its 24GB of VRAM can cache more experts. Under this workload, VRAM capacity beats raw floating-point throughput.

For Strata to work properly, the hardware needs to meet a few requirements: 12GB VRAM, 64GB RAM, a fast NVMe SSD, and a capable CPU. Insufficient RAM means the model can't load; insufficient VRAM pushes more work onto the CPU; a weak CPU turns cold experts into a bottleneck; and if PCIe transfers can't be hidden, the GPU stalls waiting.

Not Compression Magic — Scheduling Artistry

Strata doesn't squeeze 125 billion parameters into 12GB of VRAM. Instead, it reframes the problem as a scheduling challenge: sparsity reduces compute, quantization reduces storage, expert caching keeps the hot working set on the GPU, the CPU handles the rest in parallel, the SSD provides lookup services, and speculative decoding reduces the number of full passes needed.

The GPU isn't running the model alone — the whole machine runs it together. Whether this represents the future of local LLM inference or a clever workaround remains to be seen, depending on how consumer hardware and MoE architectures evolve. But it proves one thing clearly: when VRAM is the bottleneck, there's a smarter path than compressing the model — and that's orchestrating the hardware.