Strata Open Source Project: Run a 125B Model Locally on a Home PC, 5x Faster Than the Cloud

Running a 125B Model on Your Gaming PC

A 1.25 billion-parameter AI model running on an ordinary desktop computer, outputting over 250 tokens per second — it sounds like science fiction. But a free, open-source project called Strata is making it a reality.

In hands-on testing, Qwen3.8 Flash (a Mixture of Experts model) running locally output over 250 tokens per second, while the same model running in the cloud via OpenRouter managed only about 60 tokens per second. With the same prompt, the local machine completed the task in 6.6 seconds; the cloud took 33 seconds. This wasn't a fluke — in nearly every test, the local machine was 3 to 5 times faster than the cloud.

The catch: models this large typically only run on servers, because even the smallest usable version is a 66GB file that must be fully loaded into memory at runtime. The RTX 5090 used in testing has only 32GB of VRAM — nowhere near enough. Strata gets around this hardware limitation in a clever way.

How Strata Makes Your Entire PC Work Together

Strata's core idea is to involve your entire computer in inference — the GPU, CPU, RAM, and even the SSD all share the workload.

This is made possible by a property of the model itself. Qwen3.8 Flash is a Mixture of Experts (MoE) model with 512 experts per layer, but only 10 are selected for each token. So while the full model has 125 billion parameters, only about 6 billion are actually active for any single token.

Building on this, Strata uses a layered scheduling approach:

• The core components needed for every token (~3.5GB) live on the GPU;
• The remaining GPU memory stores the most frequently selected experts;
• All 24,576 experts are stored in regular RAM;
• A massive lookup table (~29GB) sits on the SSD, with only a few rows fetched per token — and prefetched in advance, so latency is barely noticeable.

Mixture of Experts (MoE) is an important architectural variant in large language models. Traditional dense models activate all parameters for every token, while MoE models replace the feed-forward layers with multiple parallel "expert" sub-networks, using a lightweight "router" to dynamically select a small number of experts per token. This allows the total parameter count to be enormous while keeping per-inference computation far below that of a comparably sized dense model. The "Flash" in Qwen3.8 Flash hints at its lightweight inference characteristics — only 10 of 512 experts are activated at a time, meaning the actual compute resembles a ~6B-parameter model while the model's theoretical "knowledge capacity" approaches the hundred-billion-parameter scale. This "large but fast" property is what makes it possible for Strata to schedule the model onto consumer hardware.

The "Doorbell" Mechanism: GPU and CPU in Sync

One of Strata's most elegant designs is its GPU–CPU collaboration.

For each layer, the GPU determines which 10 experts are needed and writes their IDs to a small shared memory location — referred to in the paper as a "doorbell." The CPU waits for this signal, and the moment the doorbell rings, it dispatches work: experts already on the GPU are handled by the GPU, while the rest are computed by the CPU directly in RAM. This lets both processors work on the same layer simultaneously, dramatically boosting throughput.

Another acceleration technique is speculative decoding. The model comes with a small draft model that guesses the next several tokens; the large model then verifies all guesses in one pass and keeps the ones it agrees with. The final output is identical, but the process runs about 1.6 to 1.8 times faster. In testing, the large model accepted the draft model's predictions roughly 79% of the time.

One notable finding from the paper: VRAM matters more than raw GPU compute — each additional GB of VRAM can accommodate roughly 700 more experts. On a 32GB RTX 5090, most experts fit in the GPU, leaving the CPU with almost nothing to do. This explains why the README reports ~94 tokens/sec for a 12GB RTX 5070, while the 5090 can exceed 250.

Installation and Hardware Requirements

Installation is straightforward: run a script, choose your model and size, download roughly 70GB of files, and a small chat app opens in your browser.

The test setup used a Core Ultra 9 285K processor, 62GB of RAM, and an RTX 5090, with the smallest and fastest quantized version selected. A few things to keep in mind:

• The computer may freeze for a few minutes on first load;
• You'll need at least 12GB VRAM, 32GB RAM, and ~80GB of free disk space;
• Once loaded, it essentially saturates the entire GPU (nearly all 32GB of VRAM in testing, plus ~39GB of regular RAM).

The bar isn't low, but it's well within reach for a mid-to-high-end gaming PC.

Local vs. Cloud: Head-to-Head Results

For a fair comparison, the tester built a small app that sent identical prompts to both local Strata and the cloud-hosted Qwen3.8 Flash simultaneously, recording speed, time, and cost.

Here are a few test results:

• Python coding challenge (find the longest palindromic substring): Strata ~4 seconds, cloud ~19 seconds; outputs were highly similar;
• Logic puzzle: Essentially a tie (1.4s vs. 1.5s) — interestingly, both fell for the classic "bat and ball" trap;
• Creative writing (product description for a secretly sentient toaster): Strata ~3 seconds, cloud ~10 seconds;
• Large task (detailed home lab server setup guide, 9,000-token limit): Strata produced over 8,000 tokens in 33 seconds; the cloud model spent the entire 9,000-token budget on reasoning and produced zero output after nearly three minutes.

On that last case, the tester honestly noted that how long a model thinks depends on the provider's settings, not the tool itself. With thinking disabled and re-run, both gave complete answers — Strata in 6.6 seconds, the cloud in 37 seconds.

On the cost side, that 33-second local response consumed less than one-tenth of a cent in electricity (though this doesn't factor in the cost of the RTX 5090 itself).

A Few Things to Keep in Perspective

The tester repeatedly acknowledged this comparison wasn't perfectly apples-to-apples, and that candor is worth noting:

• The local version runs a 2-bit quantized model; the cloud may be running a larger variant — so this is better understood as a speed test than a quality comparison;
• Cloud providers must share hardware across many users, while the local GPU serves only one person — some of the speed advantage comes from that;
• Local speed decreases slightly as conversations grow longer.

That said, even accounting for these factors, the quality of the 2-bit version's answers was broadly comparable to the cloud's in testing — which is remarkable in its own right.

2-bit quantization is an extremely aggressive model compression technique. Quantization works by mapping model weights from high-precision floats (e.g., FP16/BF16, 16 bits per parameter) to low-bit integer representations, drastically reducing VRAM usage and memory bandwidth requirements. Common quantization levels range from 8-bit and 4-bit down to 2-bit: lower bit counts mean smaller models and faster inference, but also greater precision loss, which can theoretically degrade model capability. 2-bit quantization represents each parameter with just 2 binary bits — an 8× compression over FP16 — making it the highest compression ratio in practical use today, and generally only suitable for large MoE models with inherent structural redundancy. The finding that local and cloud answer quality were "broadly comparable" partly reflects MoE architecture's strong robustness to aggressive quantization, though this should not be generalized to all model types.

Local AI Is Becoming Increasingly Practical

A 125-billion-parameter model running on a single gaming PC, substantially outpacing the cloud in speed — this would have been nearly unthinkable not long ago. Strata represents a class of engineering work that is evolving rapidly: through smarter memory scheduling, expert layering, speculative decoding, and other techniques, massive models are being squeezed into the hardware we already own at home.

As projects like this continue to emerge, running large models locally is shifting from a geek experiment to something within reach for any high-end PC user. For anyone who cares about data privacy, inference costs, or offline availability, this is a trend well worth watching.

Background

Speculative Decoding is a technique for accelerating large model inference using a smaller model, introduced by Google in 2023 and now widely adopted. The core idea: a lightweight "draft model" rapidly generates several candidate tokens in sequence, then the target large model performs a single parallel forward pass to verify all candidates simultaneously — accepting tokens consistent with the large model's distribution and resampling from the rejection point onward. Since the bottleneck in large models is typically the serial, token-by-token generation process (each step requiring a full forward pass), and verifying multiple tokens in parallel costs roughly the same compute as verifying one, overall throughput can improve significantly as long as the draft model's acceptance rate is high enough. Strata's measured 79% acceptance rate is quite strong, which is why it achieves a 1.6–1.8× speedup. Draft models are typically smaller parameter versions from the same model family as the target, to ensure similar output distributions.