Dual RTX 4090s Running Qwen3-Next: 8-Concurrent Full-Length Long-Context Benchmark

An Engineering Port Targeting VRAM Bottlenecks

Running 8 concurrent requests with context lengths approaching 256K on two RTX 4090s with 48GB VRAM each — that's the core proposition this project set out to validate. The approach ports a HiSparse-style KV offloading strategy onto QSA (the sparse attention structure used by Qwen3-Next), keeping the full history in host memory while only retaining hot data in VRAM.

The practical value of this kind of optimization is transforming long-context inference — previously bottlenecked by VRAM — from something you barely get through in a demo into an actual usable service: one that completes requests, reclaims resources, and handles multiple concurrent requests.

Worth emphasizing: throughout the original material, the author repeatedly maintains a disciplined framing. Capacity, fast-path throughput, and agent experience each have their own independent evidence — they must not be conflated, and certainly not stitched together into an inflated claim like "4x speedup." This self-restraint is precisely what makes the benchmark more credible.

QSA (Qwen3 Sparse Attention) is the attention variant used by Qwen3-Next. The core idea is that not all tokens are equally relevant to the current generation step, so the model can selectively attend to only a subset, reducing computation on long sequences. C4 blocks refer to groupings of 4 adjacent tokens into a single computational unit, with attention selection happening at the block level rather than the token level — balancing sparsity against implementation complexity. "Sink layers" are specific layers that anchor attention patterns, typically tending to attend to tokens near the beginning of the sequence. In QSA, 12 sink layers each maintain their own independent selection strategy, which is incompatible with the assumption in standard HiSparse that cross-layer shared indices can be reused — requiring per-layer independent management of index and cache slot mappings.

VRAM Ledger: Hot-Cache Offloading Saves ~84%

Start with what gets saved. Per card, a full-context raw KV for a single request is on the order of 1.5GB, so 8 requests totals roughly 12GB. After the port, 8 hot caches weigh in at approximately 396MB, plus a shared staging buffer of about 1.5GB, for a combined total of roughly 1.89GB. That's an ~84% reduction in VRAM for the KV portion, freeing up headroom for multiple long-context requests.

To be clear, this is not total card usage. Weights, indices, and GDN state still reside on-device permanently; only the "cold" portion of the KV cache is offloaded. The full history lives in pinned host memory, and missing blocks are fetched back to the GPU on demand.

The real engineering challenge lies in QSA's structural characteristics: every 4 raw tokens form a C4 block, and the 12 sink layers each select independently — so the cross-layer shared-index prefetch logic can't simply be reused. The project re-aligned raw KV indices, quantization scales, and page table layouts, and separated logical token addresses from physical cache slot management.

Beyond that, a production-ready service can't just survive a single decode pass. Requests must be completable and cancellable, with new requests safely reusing cache. The project added lease management, generational tags, and a tail ring buffer, plus an 8-way full CUDA Graph archive. Experiments show that after a request completes, KV capacity and cache slots return to their initial values — the reclamation mechanism works.

KV cache (Key-Value Cache) is the core data structure in Transformer inference that stores intermediate attention results. Each token generates corresponding Key and Value vectors at every layer, used in attention computations for subsequent token generation. The longer the context, the larger this data grows — a 256K token KV cache can easily exhaust all VRAM on a consumer GPU. The hot/cold offloading strategy is based on an observation: attention computation doesn't rely uniformly on all historical tokens. Recent tokens (hot data) are accessed frequently, while distant tokens (cold data) have low access probability. So only recent hot cache needs to stay in VRAM, while the full history is offloaded to the much larger host memory (CPU RAM), fetched back on demand. This trades PCIe transfer latency for VRAM capacity — suitable for inference scenarios that require ultra-long contexts but aren't extremely latency-sensitive. HiSparse is an earlier implementation of this idea; this project ports it to the QSA architecture.

Speed Gains: From the Fast Path, Not the Offloading

Capacity is the primary benefit, but there's also independent evidence for speed improvements — as long as you track the source correctly. In matched-configuration comparisons, single-request throughput at 38K tokens improved from 69.38 to 86.07 tokens/second, and 8-way total throughput at ~4K tokens improved from 400.16 to 464.15 tokens/second.

The author explicitly notes: these gains come from Flash Attention fast-path optimizations, not from moving KV to host memory — and they're not from the long-context stress test either. Correctly attributing gains to their actual source rather than lumping them together is the most commonly overlooked yet most critical part of evaluating projects like this.

As for the long-context archive experiment: 8 requests each with ~261,000+ input tokens, each completing ~1,022 output tokens. Total throughput over the shared decode window was 214.85 tokens/second, or ~26.86 per request. But this window was only ~2.28 seconds, and prefill was serial or interleaved — this cannot be treated as "8 streams always decoding in parallel." A full rerun hasn't been done after related fixes, so the original experimental framing is preserved here — an honest disclaimer.

Flash Attention is an algorithm that reorganizes the order of attention computations to reduce GPU memory read/write operations. Standard attention requires writing a large intermediate matrix (the Q×K score matrix) fully to VRAM and reading it back, whereas Flash Attention uses tiled computation to perform most of the work in on-chip SRAM, dramatically reducing memory bandwidth consumption. The "fast path" typically refers to a highly optimized kernel branch triggered when specific conditions are met (e.g., sequence length and head dimension fall within the expected parameter range for the optimized kernel), with a fallback to a slower general implementation otherwise. The speed improvements in this project come from modifications that triggered this fast path — they have nothing to do with KV offloading itself, which involves PCIe data transfers that theoretically only increase latency, not decrease it. Conflating gains from different sources is the most common form of result inflation in optimization projects; the author's explicit separation of attribution here reflects strong methodological integrity.

Real Terminal Logs: Interface Availability Validation

To preempt accusations of demo fabrication, the author preserved authentic terminal output captured over SSH during the production process.

After the model passes a health check on the local port, a streaming request is sent: server-side local timing shows 134ms to first chunk, ~1.6 seconds total, 130 tokens output. This short request exists only to verify the interface — the author explicitly states it "makes no claim to represent 256K performance."

Next is a more complete coding task: implement a Python streaming response parser in an isolated directory, concatenating text, preserving the last Usage entry, and stopping at the end marker. The logs show the model first reading the task and examples, then writing code, proactively refactoring a redundant branch, writing tests, running commands, and passing all 14 self-tests on the first try. Total runtime for the real task was ~49.94 seconds; an independent rerun of 14 tests plus 4 additional checks all passed with zero manual modifications.

In terms of experience: this ran in headless mode, with no text feedback visible until the very end, creating a noticeable wait. The author again emphasizes: this is evidence for a single small task only — it doesn't represent reliability on large codebases, nor is it proof of long-context acceleration.

Web and Multi-Agent Tasks: Deliverable, but Don't Overread

A web application forwarded via SSH to a local machine was used to run a few classic problems.

The first task generates an SVG of a "bird riding a bicycle," taking ~2 minutes 22 seconds from submission to completion (screen recorded at 5x speed). The raw output from the first generation is preserved unedited — the bird and bicycle are recognizable, though the helmet looks more like a visor. A relative-path error in the web preview was fixed by switching to an absolute path; the issue is documented as-is.

The second task is bracket matching. The model writes an implementation and tests but gets the expected result for an empty string wrong; it self-corrects after the failure, finishing in ~1 minute 26 seconds. On review, all 12 built-in tests passed on rerun, with additional checks covering all bracket combinations of length 0 to 6, plus Chinese command-line input and long inputs — 56,008 total checks passed.

Finally, a multi-agent scenario: 4 tasks launched simultaneously (bracket matching, topological sort, caching, interval merging), all started before receiving completion notifications, with genuine overlapping wait times.

Post-completion statistics show output speeds of ~49–52 tokens/second, with average time-to-first-token ranging from 5.7 to 10.1 seconds per task. All 47 built-in tests across the 4 programs passed on independent reruns; the caching and interval merging tasks each required corrections to test assertions.

The author draws a clear line again: these speeds cannot be summed up, cannot be described as "four streams always decoding in parallel," and cannot support a claim of 4x speedup. What this validates is simply — multi-agent tasks can be delivered.

The Real Value of This Port

Taking all these experiments together, the significance of this project is clear and measured: it transforms QSA long-context inference from "barely running" into a service that can complete, reclaim, and accept multiple concurrent requests — despite VRAM constraints.

• Capacity gains come from hot-cache offloading (~84% VRAM reduction);
• Speed gains come from the Flash Attention fast path;
• Agent experience comes from real multi-task delivery records.

Each has its own evidence and none impersonates the others. Code, patches, and experiment logs are all in the project repository. For developers who have consumer GPUs and want to run long-context and multi-agent services, this is an honest and practically useful engineering reference.