Two-Day Port, 8x Faster Prefill: Strata Engine Benchmarked

One Developer, Two Weeks, an Inference Engine 8x Faster Than llama.cpp

According to an overseas benchmark video reposted by a Bilibili creator, an inference engine called Strata runs the 125B-parameter Qwen 3.8 Flash model on a single consumer-grade GPU at roughly 900 tokens/sec prefill and about 40 tokens/sec decode. On the same card with llama.cpp, prefill is only about 120 tokens/sec. The gap is nearly 8x.

The video's author states outright that the engine is only two weeks old, MIT-licensed, mostly written by a single developer, and the repo has already accumulated around 9,000 stars. Its core bet is the exact opposite of llama.cpp's: llama.cpp supports about 150 model architectures; Strata supports one.

What does this extreme "serve only one model" trade-off buy? The answer lies in a metric most people overlook.

Why Prefill Matters More Than Decode

Most people judge an inference engine by decode speed—how fast the model writes its answer. But a real Instamart ordering case in the video reveals the other side: for an Agent to complete a single order, it round-trips the model 26 times, accumulating about 34,000 prompt tokens of input while outputting only about 1,700 tokens. For every token written, 20 tokens are read. The model spends 62% of its busy time on prefill.

This means that if your scenario is Agents, RAG, or long-context tool calling, prefill is the real source of waiting. The author's measurement is intuitive: for the same 34,000 tokens, llama.cpp at 120 tokens/sec prefill needs nearly 5 minutes, while Strata completes the same work in 73 seconds. One side is still reading while the other has already placed the order.

This judgment doesn't depend on any specific engine and applies to any local deployment choice: when picking a model or engine to run Agents, look at prefill first, then decode.

Prefill and decode are the two stages of LLM inference. The prefill stage processes the full input prompt with all tokens computed in parallel, achieving high GPU utilization—it is compute-bound. The decode stage generates tokens autoregressively one at a time, processing only one new token per step and repeatedly reading model weights and the KV cache—it is memory-bandwidth-bound. Since decode cannot be parallelized, it is usually measured in tokens generated per second, while prefill is measured in tokens processed per second. In Agent, RAG, and similar scenarios, input far exceeds output, so total prefill time often exceeds decode time.

The KV cache is the core memory overhead of the decode stage. The model stores Key and Value vectors for every processed token at every layer to avoid recomputation. With long contexts the KV cache can occupy several GB of VRAM; sparse attention reduces bandwidth pressure by keeping or reading only a subset of KV rows—this is exactly what the Qwen model's indexer does when it selects about 2,000 positions.

Where Strata Is Fast: Not Magic, but Logistics

Interestingly, Strata directly reuses llama.cpp's MMQ kernel for expert matrix multiplication—the heaviest computation in the entire model. The real gap comes from how data movement is organized.

The video uses a kitchen metaphor: 12GB of VRAM is the countertop, system memory is the pantry, and PCIe is the corridor. A 125B MoE model doesn't fit in a 12GB card, so most experts must stay in memory. The bottleneck isn't the chef's skill—it's how many times ingredients travel through the corridor.

Strata's approach is to increase the batch size per trip. llama.cpp processes prompts in batches of 512 tokens by default, moving non-resident experts from memory once per batch; Strata borrows nearly all available VRAM as cache during prefill and chunks at 6,144 tokens, dropping the frequency of experts crossing PCIe from once per 500 tokens to once per 6,000. It also uses a separate thread to DMA the next layer's experts into a GPU ring buffer ahead of time, so while layer N+1 is loading, layer N is still computing—the GPU no longer idles waiting on copies.

Attention is another difference. The model uses sparse attention, with an indexer selecting about 2,000 positions per query. Upstream llama.cpp still builds a full mask and runs dense flash attention, whereas Strata uses its own kernel to read only the selected KV cache rows.

The decode stage has a different problem: there's no big batch to hide behind—every token is a brand-new journey. Strata splits each layer into three parallel paths—the GPU handles cache hits, about a quarter of misses are pulled over PCIe, and the rest are computed directly by the CPU from pinned memory, all three proceeding simultaneously. Add the model's built-in MTP multi-token prediction head, which drafts up to three tokens per step and verifies them with the full model in one pass, producing output identical to running without speculation. The author's tuned llama.cpp build decodes at about 26 tokens/sec on this card; Strata reaches about 40.

The MMQ (Mixed-precision Matrix Multiplication) kernel is the core of llama.cpp's quantized matrix multiplication. It supports low-precision (e.g., 4-bit, 5-bit) block quantization of weights at inference time and dynamically dequantizes during the matrix multiply, enabling large models that exceed VRAM capacity to run on consumer GPUs. Strata reusing this kernel for expert matrix multiplication shows that its performance advantage comes not from optimizing compute instructions but from data scheduling.

The MoE (Mixture of Experts) architecture is key to running a 125B-parameter model on a single card. Each layer contains multiple expert networks (512 in the figure), and each token activates only a few of them (e.g., 10), so actual computation is far less than total parameter count. But all expert weights must still be stored; the portion that doesn't fit in VRAM stays in system memory and is moved over PCIe on demand. The larger the prefill batch, the more tokens each expert is reused for, and the lower the movement cost amortized per token—this is the logical basis for Strata's larger batches.

Porting Experiment: Two Days, Prefill from 120 to 620

The author did something more interesting: after reading Strata's code, he spent two days porting the prefill ideas into his own llama.cpp branch—borrowing the expert cache for large-block processing, pre-streaming expert loads on a second CUDA stream, and adding a dedicated kernel for sparse attention. On a 17,000-token prompt, his branch improved from about 120 tokens/sec to about 620, while Strata measured about 900 on the same test.

He didn't beat Strata, but he reached a more important conclusion: writing an extreme kernel for one model is far easier than building a general-purpose tool. Moreover, about two-thirds of Strata's commits are co-authored by AI coding Agents. This means the speed of writing dedicated kernels for new models is becoming extremely fast.

His prediction is that two kinds of engines will coexist: one that runs a single architecture extremely fast, and general engines that sacrifice some speed to cover most models. Over time, specialized kernels will be gradually absorbed into general engines, then new models appear, and the cycle repeats.

A CUDA stream is an asynchronous execution queue on the GPU. By default all operations execute sequentially in the same stream; creating a second stream allows data copies to overlap with computation. "Pre-streaming expert loads on a second CUDA stream" exploits exactly this mechanism: while the main stream computes layer N, the second stream DMAs the experts needed for layer N+1 from host memory into a GPU buffer, hiding PCIe latency. This technique matches Strata's ring-buffer approach and is one of the key means behind the porting experiment's 5x improvement.

AI-coding-Agent co-authored commits refer to code commits generated or assisted by tools like Claude Code and Cursor. About two-thirds of Strata's commits carry such attribution, indicating that the marginal cost of writing dedicated kernels for a single model is falling rapidly—this changes the pace of competition between general and specialized engines.

Deployment and Three Pitfalls You Must Know

Strata provides an OpenAI-compatible server but serves only this one model. Other models remain as presets in llama.cpp's router. The author uses LamaSwap as a front proxy to launch the corresponding backend by model name. A request for the 35B launches the router; a request for Qwen Flash stops the router and starts Strata—switching takes about 34 seconds on a single card, with responses in about 15 seconds. After 10 minutes idle, Strata shuts down automatically and releases pinned memory.

The first pitfall is obvious: Strata serves one request at a time, with no batching. For multi-user concurrency or running multiple Agents in parallel, it's not a suitable choice.

The second pitfall is memory pinning. The author's IQ3-quantized version pins about 40GB of fixed memory that other programs on the machine cannot use while Strata runs. Under Windows there's a cap on multi-GPU pinning, WSL allows even less, and Linux is where it truly belongs.

The third pitfall concerns output consistency. Whether the same expert is computed by GPU or CPU produces tiny numerical differences, and the cache constantly moves experts between the two, so even with greedy decoding it is not byte-for-byte reproducible by default. A few switches can force consistency at the cost of speed. The author calls this noise-level—Strata's paper measured its output distribution difference from llama.cpp as smaller than llama.cpp's own difference between CPU and GPU.

One setting deserves a separate warning: experimental speed projection, off by default. The name sounds like an acceleration feature, but it's actually a refusal-removal control vector. When enabled, about 1 in 10 positions changes its top token, and code perplexity rises 15%. Keep it off.

The MTP (Multi-Token Prediction) head is a lightweight module attached during model training that lets the model draft multiple candidate tokens at once, which the full model then verifies in parallel, accepting results consistent with greedy decoding. This is a form of speculative decoding that raises decode throughput without changing the output distribution. Strata uses the model's built-in MTP head to draft up to three tokens per step, one reason its decode speed rises from about 26 to about 40 tokens/sec.

A refusal-removal control vector is a direction vector some models embed during training to suppress certain outputs at inference. The experimental speed projection switch, named for acceleration, actually disables the removal of this control vector, shifting the output distribution and raising perplexity. Perplexity measures a model's predictive uncertainty over text; a 15% increase means a real drop in generation quality.

Can It Replace Your Daily Driver Model?

The author's personal answer: it already has. He previously used a 35B MoE as his daily driver because it was fast rather than smart. Qwen 3.8 Flash, in his view, is on another level—close to frontier. And Strata speeds up the prefill stage where Agents actually wait; that Instamart order completed in under two minutes on a single card.

His recommendation is tiered by memory: with 64GB, the full model fits and you can switch directly; with only 32GB, try the expert-pruned Coder version, suited to coding-focused Agents; and with LamaSwap in front, you don't have to choose—the 35B can stay.

The cold water is also clear: the project is only two weeks old, most published data comes from a single machine, and the docs themselves admit this and label their estimates. Problems between versions are to be expected.

One-line Summary

Strata, built by a single developer in two weeks, uses specialized memory scheduling to make MoE prefill 8x faster than llama.cpp.

Paragraph Summary

The article introduces Strata, an open-source inference engine only two weeks old and led by one developer. It runs the 125B-parameter Qwen 3.8 Flash model on a single consumer GPU at about 900 tokens/sec prefill—8x llama.cpp. The core reason is that Strata supports only one architecture, enabling extreme data-movement optimization: larger prefill batches, pre-streamed expert loading, and a dedicated sparse-attention kernel. The article notes that in Agent scenarios prefill, not decode, is the main source of waiting, and documents the author's two-day port that raised a llama.cpp branch's prefill from 120 to 620 tokens/sec. It also warns that Strata lacks concurrency, pins large amounts of memory, and is not byte-for-byte reproducible.