Running a 125B Coding Model on a Single RTX 5090: Throughput Bottlenecks and Tuning Notes

Replacing your team's per-seat AI coding subscriptions with a single consumer GPU sounds tempting. But what really determines the experience isn't benchmark scores—it's "how badly things slow down when several people use it at once." This article is based on measurements from Strata (an open-source engine that runs the Qwen3.8-Flash-Next 125B MoE model on consumer GPUs). We break down the throughput patterns of a single card serving multiple users, two hidden performance traps, and what it's actually like to hook it up to Claude Code.

Start With the Numbers You'll Actually Use

The test environment was a rented RTX 5090 (32GB VRAM) running IST Austria DASLab's 1-bit Coder quantized version (58.4GB), with a 256K context.

• 1 developer: 144 tokens/s, first-token latency under 2 seconds
• 4 concurrent users: 38 tokens/s each, 125 tokens/s total
• 8 concurrent users: 19 tokens/s each, 130 tokens/s total, first-token latency 2 to 8 seconds

The key pattern: total throughput barely grows with the number of users—it stays flat. What this card runs out of is "concurrent seats," not "token production capacity." Understanding this is essential to planning how your team should use it.

Why One Person Is So Much Faster Than Four

When a single user is active, the engine enables speculative decoding: a small draft model guesses 4 tokens ahead, and the large model verifies them all at once—about three-quarters of the drafts were accepted in testing. This is a major source of the 144 tokens/s figure.

But when multiple users are concurrent, Strata uses a parallel slots mechanism, and these slots do not use speculative decoding—each token is generated and emitted one at a time. So when you go from 1 user to 4, per-user speed drops from 144 to 38, and total throughput declines somewhat too—a bigger loss than simply "splitting the card."

The practical takeaway is straightforward: if your team members' generation requests are staggered and rarely overlap, the single-card experience is quite good; if you're running multiple agents in parallel all day, do capacity planning based on the concurrent-slot numbers, not the single-user peak.

Speculative Decoding is a speculative technique for accelerating large-model inference: first, a much smaller "draft model" rapidly guesses several tokens in sequence, then the target large model verifies that string of drafts in parallel during a single forward pass. Any part where the draft matches the target model's output is accepted directly, and sampling only needs to restart from the first mismatch. Because verification is parallel, a single large-model forward pass can produce multiple tokens, breaking past the speed ceiling of token-by-token autoregressive decoding. Its benefit depends on the "acceptance rate"—the more accurate the draft, the greater the speedup; the roughly three-quarters acceptance rate measured here is precisely the main source of the 144 tokens/s single-user figure. The cost is that you need to load an extra draft model, which consumes VRAM, and it's hard to combine with multi-request scheduling in batched concurrent scenarios—which is why parallel slots don't enable it.

Parallel slots are how an inference engine schedules multiple concurrent requests: the KV Cache in VRAM is divided into a number of fixed slots, each holding an independent session, and the engine uses continuous batching to merge the decoding steps of all slots at a given moment into a single batched computation. The advantage is that VRAM and compute are shared stably when multiple people use it at once, and total throughput doesn't collapse as users increase; the disadvantage is that every slot follows the standard token-by-token decoding path and can't layer on single-request-specific accelerations like speculative decoding. So the number of slots directly determines "how many people can be served at once," while per-slot speed falls as concurrency rises—this is the mechanistic root of the flat total throughput and sharp drop in per-user speed.

The 5x Slowdown That Wasn't the GPU's Fault

The first rented host only managed about 30 tokens/s when ~170 was expected. The GPU itself was fine. The problem was that the container exposed all 256 threads of the host machine, but a cgroup quota limited the CPU to about 30 cores. Strata sized its worker pool based on the "visible thread count," the workers spun idle and got throttled by the kernel, and CPU-side work (expert layers not cached on the GPU) became the bottleneck.

Before running any benchmark on a rented container, check:

cat /sys/fs/cgroup/cpu.max

If it shows a quota limit, set --pool-workers to a value slightly below the allowed core count, and generation speed immediately returns to normal.

The second trap appeared at 8 slots: with the default configuration, less than 400MiB of VRAM remained, and the driver began paging VRAM. Adding --vram-reserve-mib 1536 fixed it. Neither problem was the model's or the GPU's fault—it was deployment parameters not matching the runtime environment.

cgroup (control group) is a Linux kernel mechanism for limiting and isolating the resource usage of process groups; containers (Docker, Kubernetes, etc.) rely on it to constrain CPU, memory, and other quotas. The key distinction here is that the "visible thread count" is determined by the host topology—inside the container, nproc or /proc/cpuinfo often still show all of the host's cores—while "available compute" is determined by the cgroup's CPU quota (the quota/period ratio in cpu.max). When a program creates a large number of worker threads based on the visible thread count while the actually available CPU time is far less than that, the threads frequently spin idle and get forcibly throttled by the kernel, manifesting as a cliff-like drop in overall performance. The way to diagnose this is to read /sys/fs/cgroup/cpu.max: if the first column isn't max, a quota limit exists, and the worker pool size should be set based on that rather than on the thread count.

Pointing Claude Code at a Local Endpoint

Strata exposes an Anthropic-compatible endpoint, so Claude Code only needs a base URL and a token:

export ANTHROPIC_BASE_URL=http://127.0.0.1:8080
export ANTHROPIC_AUTH_TOKEN=local

In a single-file task (a Snake game with an autopilot mode), the model wrote 408 lines of code in one go and fixed a timing bug on its own. One notable quirk: these heavily quantized models sometimes claim they've finished without actually calling the Write tool. Naming the Write tool explicitly in the prompt made the problem disappear.

The Limits of These Numbers

The limitations deserve an honest disclaimer: only one card, one host, one single-file task (not a real code repository). The quality metric of 75.6% (SWE-bench Verified) comes from the model card, compared with 82.8% for the full model—it was not measured in this article.

For teams considering replacing per-seat subscriptions with a self-hosted GPU, the real positioning of this card is: single-user experience approaches commercial services; multi-user concurrency is a capacity arithmetic problem. Get the two parameters—cgroup quota and VRAM reserve—right first, then talk about cost-effectiveness.

MoE (Mixture of Experts) architecture splits the feed-forward layers into multiple "expert" subnetworks, and each token is routed to only a few of them for computation. This allows the model's total parameter count to be very large while the number of parameters actually activated per inference is far smaller than the total—which is exactly the case for the 125B model in this article, and why it can run on a single 32GB consumer GPU. But it also brings a deployment characteristic: the weights of unactivated experts still need to reside in VRAM or be swapped in quickly, and when the number of concurrent slots grows and the KV Cache squeezes VRAM, expert layers may be forced onto the CPU side or trigger paging, significantly slowing things down. The CPU-side expert-layer bottleneck and the VRAM paging issue mentioned in this article are both directly related to this "large total parameters, small activated amount" property of MoE.

One-line Summary

A single RTX 5090 running a 125B coding model: total throughput stays constant—the bottleneck is concurrent seats, not compute.

Paragraph Summary

Based on measurements of the Strata engine running the Qwen3.8-Flash-Next 125B MoE model on a single RTX 5090, the article presents the throughput pattern for multiple users on one card: 1 user at 144 tokens/s; 4 users at 38 each, 125 total; 8 users at 19 each, 130 total. Total throughput barely grows with user count, and the real bottleneck is concurrent seats rather than token production capacity. The high single-user speed comes from speculative decoding, which the multi-user parallel slots don't enable, causing per-user speed to plummet as you scale. The article also reveals two hidden traps: a container cgroup limiting CPU cores while exposing all threads, causing roughly a 5x slowdown; and insufficient VRAM at 8 slots triggering driver paging. Finally, it covers how to configure Claude Code and the quirk of quantized models skipping the Write tool, while candidly acknowledging the limited scale of testing.