RTX 3090 Tested: Qwen's 381 tok/s Myth Busted, Real Chat Only Hits 118

A banner claiming 381 tok/s on a single RTX 3090 made the rounds in tech circles last month. A user with the same GPU personally reproduced this inference stack called HyperQwen, only to find: real chat throughput is just 118 tok/s, and that tempting 381 figure not only depends on non-default settings but also crashes the engine outright on long-context requests.

What Exactly Is This Stack

HyperQwen was released by GitHub user syv-ai — an Apache-2.0 vLLM custom build specifically tuned for the Qwen3.8-27B model in W4A16 quantization format, with weights occupying about 19.5GB and DFlash2 speculative decoding onboard. The comparison baseline is a Q4_K_M GGUF version of the same 27B model running on llama.cpp, at roughly 42 tok/s on the same card.

The original post claimed an uplift from 82 to 381 tok/s, but after actual testing: 82 is the starting baseline of vanilla vLLM, and 381 only appears under citation-repetition workloads. That figure requires setting DFLASH_TOKENS=15 — a non-default, longer draft block parameter. After trying it, the author found chat performance unchanged, but citation requests at 20k context crashed the engine directly, throwing EngineDeadError and marlin_gemm allocation failures. Afterward every request returned 500, and even after container restart it could keep returning 500 with no self-recovery.

W4A16 quantization refers to a mixed-precision scheme where weights are stored in 4-bit and activations remain 16-bit. Compared to GGUF's commonly used Q4_K_M (also 4-bit weights), W4A16 retains a higher-precision activation path, and with dedicated CUDA kernels like marlin it can achieve higher throughput on NVIDIA GPUs — at the cost of more rigid VRAM usage and stricter kernel support requirements. The marlin_gemm allocation failure mentioned in the article is exactly the kind of failure unique to this path.

Speculative decoding uses a small draft model or draft block to predict several upcoming tokens, then the main model verifies them in batch and accepts the matching portion. It yields the biggest gains in deterministic scenarios like greedy decoding, and shrinks noticeably in sampling or thinking-intensive scenarios. The longer the draft block length (such as DFLASH_TOKENS), the greater the theoretical speedup, but verification failure rates and VRAM overhead rise accordingly.

Draft-flash-style approaches (like DFlash2) are designed to let the draft stage reuse part of the main model's computation rather than relying entirely on a separate small model, making them more sensitive to KV cache and VRAM pressure under long contexts — which is precisely the trigger for the 20k-context crash described in the article.

Real Chat Performance Benchmarked

On thinking-intensive prompts with greedy decoding (the worst case for speculative decoding), four prompts averaged 118.1 tok/s, ranging from 111 to 131. At temperature 0.7 sampling it was 123.3 tok/s. The project's own single-user chat figure is about 127 tok/s, so real-world testing falls within a reasonable range.

In the context-citation scenario — a 29k token prompt having the model verbatim reproduce a passage — decoding reached 237.6 tok/s with verbatim verification passing. Prefill was about 1,210 tok/s, completing that prompt in about 24 seconds, roughly five times llama.cpp (~250 tok/s). From startup to healthy was about 3 minutes (including first JIT compilation), and about 1 minute after warmup.

Compared to the GGUF baseline's 42 tok/s: real chat is three times faster, citation scenarios six times faster. Test conditions were a 420W power limit, driver 610.57, CUDA 13.3.

Prefill and Decode are the two phases of inference. Prefill processes the entire prompt in parallel at once and is compute-bound, so its throughput in tok/s is usually much higher than decode; decode generates tokens autoregressively one at a time and is limited by memory bandwidth. In the article, prefill is about 1,210 tok/s while decode is only in the hundreds of tok/s — a typical asymmetric performance profile on a consumer single card, which is why for long prompts the "time to first token" often affects perceived responsiveness more than total throughput.

Greedy decoding takes only the highest-probability token at each step, producing deterministic output, and is the standard scenario for measuring the upper bound of speculative decoding; once temperature sampling introduces randomness, the probability of drafts being accepted drops and the speedup ratio shrinks accordingly. This explains why the article's 123.3 tok/s at 0.7 sampling is so close to the 118 tok/s under greedy decoding.

Two Easy-to-Fall-Into Measurement Traps

In the first round of testing, the author counted streaming chunks and got 43 tok/s — almost one third of the actual 118 tok/s, which would make this stack look slower than the GGUF it replaced. The reason is that speculative decoding changes the shape of streaming output: vLLM does not send one token per SSE chunk, but sends the entire accepted batch, which with 7 drafts is about 2.5 tokens per chunk. The correct approach is to read usage.completion_tokens from the final chunk, adding stream_options: {"include_usage": true} to the request.

The second trap: the model puts its thinking process under delta.reasoning rather than reasoning_content, and it thinks on every prompt. max_tokens must leave enough headroom, otherwise the thinking budget eats into the answer.

In the SSE streaming protocol, the server usually sends one data: chunk per token, so counting chunks approximates the token count. But vLLM, after batch-accepting drafts, merges the whole batch into a single chunk, so chunk count and token count no longer correspond one-to-one. The article's roughly 2.5 tokens/chunk stems from this. A more robust approach is to parse usage.completion_tokens in the final chunk, which is authoritatively counted server-side and unaffected by streaming packaging, requiring stream_options.include_usage to be explicitly enabled in the request body.

The dual-field problem of thinking models: different frameworks do not use consistent field names for chain-of-thought (reasoning, reasoning_content, thinking, etc.), and thinking content also counts toward the max_tokens budget. If a client only reads the answer field and ignores the thinking field, or lets thinking get truncated when the budget is insufficient, you get false failures like "empty answer" or "irrelevant response," easily misdiagnosed as inference stack performance problems.

The Physical Laws of a 24GB GPU

The author summarized three operational rules, each earned through an afternoon of debugging.

The server must be warmed up first. The compose entry defaults to WARMUP=0, and the first batch of requests triggers kernel compilation for the serving path, making cold-start numbers completely unreliable. Run bench/warmup.sh after every startup.

Frequent agent cron jobs must be paused. llama-swap burns a 420-second health check timeout per request on dead members, and cron agents will error out mid-run.

The most critical one: systemctl stop strata will also stop llama-swap, because llama-swap carries Requires=strata.service. And llama-swap cannot load GGUF members while Strata is running, because Strata always holds VRAM. The HyperQwen window can only bypass the proxy and connect directly to :18020, at which point :8080 goes offline. A single 24GB card can only run one large model at a time — that's physics; proxy configuration just makes it visible.

JIT compilation and warmup: vLLM compiles CUDA kernels on the fly the first time it encounters a given shape or dtype combination, so the first request after cold start has extremely high latency and throughput of 0 or very low. The purpose of a batch warmup script is to trigger these compilation paths in advance so steady-state numbers are reproducible.

llama-swap-style proxies "share" a GPU by starting and stopping member models on demand, but they cannot bypass VRAM exclusivity: as long as the resident model still occupies VRAM, the candidate model simply cannot be loaded. systemd's Requires= is a hard dependency — stopping the depended-upon service also stops the dependent one — so the intuition of "stopping just one service" does not hold under chained dependencies. On a 24GB card, the feasible combination of a single large model plus a proxy is essentially a VRAM budget allocation problem, not a software configuration problem.

What's Running Now

In steady state, HyperQwen has been shut down. Strata has returned to resident service, and after verifying with real completions, all seven cron jobs are back, with the Windows watcher cron from the first attempt officially retired. The wiring to turn HyperQwen into an on-demand llama-swap member has been written and staged, awaiting the routing decision.

Before chasing the number 381, it's advisable to bring your own actual running workloads. The citation path is indeed very fast, and chat is roughly three times the GGUF path, but the setting that supports the headline number happened to crash the author's engine at exactly the long-context size the headline required.

One-line Summary

HyperQwen tested on RTX 3090 delivers 118 tok/s in real chat, far below the advertised 381 tok/s.

Paragraph Summary

The author reproduced the HyperQwen inference stack on an RTX 3090 (a vLLM-based W4A16 quantization build for Qwen3.8-27B with DFlash2 speculative decoding), finding that the original post's claimed 381 tok/s only appears in citation-repetition scenarios and requires enabling the non-default parameter DFLASH_TOKENS=15, a setting that triggers EngineDeadError at 20k context and causes subsequent requests to persistently return 500. In real chat scenarios, greedy decoding averages 118 tok/s, about three times the GGUF baseline of 42 tok/s; citation scenarios hit 237.6 tok/s, about six times the baseline. The article also points out two measurement traps (counting by SSE chunk underestimates by about 2.5x, and thinking field naming plus max_tokens budget) and the VRAM exclusivity physics of a 24GB GPU, ultimately leading the author to shut down HyperQwen and return to Strata as a resident service.