12GB VRAM Running a 180B Model? A Deep Dive into the Strata Inference Engine

A standard desktop PC with an RTX 5070 (12GB VRAM) and 64GB of system RAM, running a model whose main files alone weigh in at 37.6–47GB plus a 28.8GB standalone lookup table — it sounds like science fiction. Yet Strata, a newly appeared inference engine on GitHub, claims to deliver 44.8 tokens per second at 128k context on exactly this hardware. Is that number real? Does the underlying architecture actually hold up?

Strata is an inference engine built by a solo developer to run the Qwen 3 series of large language models. A Reddit thread discussing it even floated the claim of being "3× faster." This article, based on a teardown analysis by a Bilibili creator, walks you through the genuinely clever design ideas — and the parts that don't survive scrutiny.

The Model Size Question: 125B or 180B?

There's conflicting information online — some sources say 125B, while the Hugging Face model page lists approximately 180B. Both numbers are correct; they just answer different questions.

The model card's spec sheet is explicit: 125 billion parameters for the core model, plus a 51 billion-parameter n-gram lookup table, plus 4 billion parameters for a draft layer called MTP (Multi-Token Prediction). That adds up to 180B.

In other words, 125B is the part that actually "does the thinking"; 180B is the total when all components are counted. When reading posts about this model, pay attention to which number is being cited, because these additional components end up living in completely different "rooms" within the system.

Mixture of Experts: Why a Small Kitchen Can Handle the Load

This model uses a Mixture of Experts (MoE) architecture internally. Its 48 layers each contain 512 small expert networks — multiplied together, that's 24,576 experts in total, which is exactly what Strata manages.

The reason a small kitchen can handle this load is sparse activation: for each token processed, the router in each layer selects only 10 out of 512 experts, plus one shared expert that runs continuously across all 48 layers. That works out to 480 routed experts per token — and in terms of parameter count, only roughly 6B out of 125B are activated.

Twenty-thousand-plus dishes on the menu, but any given order only touches about 2% of them. You never need to lay out the entire menu at once — you just need to pull the right 2% quickly.

A note on MoE architecture: The core idea behind Mixture of Experts is to split a large neural network into many smaller "expert" sub-networks, activating only a small fraction during each inference pass. This contrasts with traditional dense models, which engage all parameters for every token. The router is essentially a lightweight classification network that scores each token and selects the best expert combination. The upside: total parameter count can scale very large (covering more knowledge in theory), while actual compute (FLOPs) scales only with the activated parameters. Qwen 3 activates roughly 6B out of 125B, meaning the compute per inference pass is close to a 6B dense model — but the knowledge capacity approaches 125B. The tradeoff is memory pressure: all expert weights must remain accessible, so you can't simply load only "what's needed for inference" the way you might with a dense model. Strata's multi-tier storage scheduling exists precisely to resolve this tension.

Four Storage Tiers: VRAM, RAM, and SSD Each Playing Their Role

Strata's most elegant design decision is distributing different parts of the model across different storage tiers — like a kitchen with a prep counter, a pantry shelf, and a basement storeroom.

VRAM (the countertop): Everything needed for every token lives permanently in VRAM — attention layers, a layer type called delta net, routers, the shared expert, the output head, and the KV cache holding 128k of conversation context. These are the knives and pans you grab every single time.

Expert cache (the nearby shelf): Strata carves out roughly 6GB of VRAM as an expert cache, holding about 4,500 experts — fewer than one-fifth of the total. When a router selects an expert that happens to be on the shelf, the GPU handles it locally.

System RAM (the storeroom): All 24,576 experts are pinned in system RAM. The critical design insight: when a router selects an expert not in the VRAM cache, Strata doesn't haul the expert weights down the PCIe 4.0 "hallway" (roughly 26 GB/s) to the GPU. Instead, it has the CPU — a "second chef" — perform the computation directly in RAM. Meanwhile the GPU continues processing other parts of the same token, and both results land on the same order.

This CPU-GPU co-computation is the core mechanism for avoiding the bottleneck of slow data transfers. But the second chef isn't free — Strata uses several compressed, quantized variants. The IQ3_XXS version that produced the 44.8 token/s result compresses weights to roughly 3 bits; Q2 and IQ2 variants go down to roughly 2 bits. Even so, IQ3_XXS's experts alone occupy roughly 43GB of RAM, plus another ~6GB — and your 64GB is largely consumed before you open a browser.

A note on PCIe bandwidth: PCIe (Peripheral Component Interconnect Express) is the high-speed bus connecting the GPU to the CPU and memory. PCIe 4.0 x16 has a theoretical bandwidth of roughly 32 GB/s, with usable bidirectional bandwidth typically around 26 GB/s. That sounds generous, but it's extremely tight in a scenario where expert weights must be transferred per token: with each expert weighing a few MB, selecting 10 experts per layer across 48 layers means potentially gigabytes of data movement per token — far beyond what PCIe can sustain in real time. This is precisely why Strata chooses "CPU computes in RAM" over "transfer weights to GPU" — it bypasses the PCIe bottleneck, trading CPU cycles for transfer bandwidth. The cost is significantly higher CPU load, and quantization formats like IQ3_XXS introduce precision loss that affects the quality of CPU-side computation.

The 51B Lookup Table in the Basement

The 51 billion-parameter n-gram table is kept on SSD. An n-gram is a short consecutive sequence of tokens; this table stores learned entries for them — essentially a 28.8GB phrase dictionary of enormous proportions.

It can live on the slower SSD because of how it's used: it's a lookup table. There's no heavy computation across the entire table — just flip to a row, read it, close it. Strata's diagram shows 16 row reads per token from the NVMe drive, issued one step ahead so the page is already in transit before the kitchen asks for it.

Then there's the 4B-parameter MTP layer — the "sous chef" for speculative decoding. It quickly predicts up to 3 upcoming tokens; the full model then verifies these guesses in a single pass, accepting them in order until it hits one it disagrees with. Strata reports an average of 2.4 to 3.2 accepted tokens per verification, meaning one expensive full-model inference can produce two or three dishes per pass.

A note on speculative decoding: Speculative decoding is a technique for accelerating autoregressive language model inference. In standard inference, generating each token requires a full pass through the large model — throughput is bottlenecked by this serial structure. Speculative decoding introduces a lightweight "draft model" that quickly proposes several tokens in a row, which the full model then verifies in a single parallel pass. Since parallel verification costs far less than serial generation, overall throughput improves. The key metric is acceptance rate: the better the draft model's guesses, the more tokens each full-model pass confirms. Strata's reported 2.4–3.2 average accepted tokens suggests reasonable draft quality. Worth noting: speculative decoding offers limited improvement to latency (time to first token); its main benefit is throughput, and it pays off most in long continuous generation scenarios.

Auditing the Numbers: How Credible Are 44.8 and "3× Faster"?

Strata published a benchmark speed table for the RTX 5070 — the most valuable artifact in the entire repository, because the numbers circulating online don't agree with each other.

At 128k context, IQ3_XXS delivers 44.8 tokens/s; the row above it, IQ2_XS at 2-bit, shows 52; and Q2_0 at 2-bit shows 65.1. You may have seen "~65 tokens/s" cited — two cells in this table match that: Q2_0 at 128k (65.1), or IQ3_XXS at 4k short context (65.6). So 65 is a real number — it just answers a different question.

For 128k context with the IQ3_XXS configuration, the correct figure to cite is 44.8.

As for the "3× faster" claim: the baseline is stated as 15 tokens/s, and 44.8 ÷ 15 ≈ 3. But that 15 doesn't appear anywhere in the repository documentation, and there's no indication of what software or configuration it refers to. "3× faster than an unknown baseline" tells you far less than it sounds.

More importantly, every speed figure in this table is self-reported by the developer — not a single one has been independently verified. Numbers for other GPUs (like the 5060 Ti or 3090) aren't even measured; the documentation explicitly describes them as estimates with ±20% error margins. And IQ3_XXS being labeled "best" refers only to speed — it says nothing about answer quality. Don't switch to a 2-bit version just because it's faster without understanding the quality tradeoff.

A License Field Left Blank

Before you run git clone, there's one more thing to check: Strata's repository was published just days ago, and the license field is empty. It builds on llama.cpp and ggml under the hood (both MIT-licensed), but Strata's own code carries no license — you can read it, but you have no right to reuse it.

A note on missing licenses: In open-source software, the absence of a license is not a minor detail. Under default copyright law, code without a license means "all rights reserved" — anyone can view it, but copying, modifying, distributing, or using it in a product without explicit author authorization constitutes infringement. This is in sharp contrast to permissive licenses like MIT or Apache 2.0. Since llama.cpp and ggml (Strata's dependencies) are MIT-licensed and permit commercial use and modification, the missing license on Strata's own layer leaves the legal status of any derivative work ambiguous. For personal research or learning, the risk is low. For internal enterprise tools, commercial products, or redistribution, you'll need to wait for the author to specify licensing terms. Missing licenses on freshly published projects are a common oversight that developers often address quickly — but until then, observe with caution.

Conclusion: Clever Design, Unverified Numbers

If you have a 12GB GPU, 64GB of RAM, and enjoy reading engine internals, Strata is worth an evening of study. Its "kitchen layout" is genuinely clever — four storage tiers each with a defined role, CPU-GPU co-computation, speculative decoding acceleration. The overall logic coherently explains how 44.8 tokens/s could be achievable on a developer's desktop.

But treat 44.8 as the developer's claim, not a settled benchmark. Know that choosing IQ3_XXS means leaning heavily on your CPU. And don't build anything on top of this until there's a proper license. The real test belongs to you: when others run this architecture on their own RAM and their own SSDs, will 44.8 survive?