4x R9700 Benchmark: Qwen3.8 Long-Context Concurrent Inference Performance

A Reddit user in r/R9700 shared a complete benchmark of his self-built 4-GPU Radeon AI PRO R9700 inference server. This system is no benchmark toy—it is a business-oriented multi-user inference server that handles sensitive customer data. He previously relied on unified-memory devices like the Strix Halo 395+ with 128GB, and has now finally moved the workload to a discrete GPU cluster.

A few numbers are worth putting up front: single-user write speed of about 190 tok/s; prefill total throughput that stays stable at about 6,800 tok/s regardless of how many users are concurrent; with 8 users each at 128K context, follow-up turns that hit the cache get their first token in just 0.2 to 2.5 seconds, with per-user writes of 80 to 225 tok/s. The worst case is 8 users simultaneously sending brand-new 128K prompts, where everyone waits about 2.5 minutes for the first token.

Hardware Configuration and Cost

The overall platform is quite pragmatic: an EPYC 7402P processor, an Asrock ROMED8-2T motherboard, 128GB of DDR4 2666 memory, three Gigabyte R9700s plus one XFX R9700, all power-limited to 210W. The power supply is a Seasonic Prime TX-1600W, and the case is a Phanteks Enthoo Pro 2 with three 140mm fans in front, three 120mm fans in the middle, and one 120mm exhaust fan in the rear.

The author specifically notes that the four cards are not all the same brand. As a perfectionist he is slightly unhappy about this, but once installed in the case he doesn't care. The CPU, memory, and motherboard were all bought on eBay, and the whole setup cost about CAD 16,000. Cooling performance exceeded expectations—at full load it is quieter than his air purifier on its highest setting, with no jet-engine vibes at all.

Software Stack and Model Deployment

The system runs Ubuntu 26.04.1, using the amdgpu driver built into kernel 7.0 and ROCm 10.1. The inference framework is Radiance v1.17.0, with the Docker image pinned by digest. The model is qwen3.8-next-flash-fp8-iq4r-moe, with an int8 backbone plus 4-bit codebook experts, featuring MTP speculative decoding (3 tokens).

Under the tp4 configuration, the entire model resides permanently in VRAM, with 17.4 GiB of weights per card. The 47.7 GiB n-gram/PLE table is pinned in host memory, and each card has 11.6 GiB of KV cache, totaling 1.6M tokens—equivalent to 8 full 200K sessions. Startup parameters include --tp 4 --max-num-seqs 8 --max-model-len 200000 --placement expert_tiered, the KV cache uses fp8 format, and speculative decoding is 3 tokens.

tp4 (tensor parallelism): Splits the weights of a single model across 4 GPUs for cooperative computation, one of the multi-GPU parallelism strategies used by inference frameworks such as vLLM/SGLang. Unlike pipeline parallelism, which splits by layer, tensor parallelism splits each weight matrix itself, so every layer's computation requires cross-GPU communication (all-reduce) and is sensitive to inter-GPU interconnect bandwidth. This is why the article emphasizes "the entire model resides permanently in VRAM, with 17.4 GiB of weights per card"—under tp4 no single card has enough VRAM to hold the full model, so high-speed interconnect is essential.

MTP speculative decoding: Multi-Token Prediction, where a small prediction head built into the model drafts multiple candidate tokens at once, which the main model then verifies in parallel, outputting multiple tokens at once if accepted. The "3 tokens" in the article means up to 3 draft tokens accepted per step, which improves decode throughput without changing model quality, and is especially suited to VRAM-bandwidth-constrained single-GPU scenarios.

n-gram/PLE table: PLE (Per-Layer Embedding) maintains a separate embedding for each layer, while the n-gram table caches high-frequency word sequences to speed up or stabilize generation. Both are enormous (47.7 GiB) and sparsely accessed, so they are pinned in host memory rather than VRAM, prefetched on the CPU side, avoiding squeezing KV cache space.

Test Methodology and Results Interpretation

The tests use real English technical text (Radiance's own documentation), tokenized to exact token counts with the server's tokenizer. Each user's prompt begins with a unique session line to eliminate cache reuse. All users in the same round issue requests simultaneously, constituting a worst-case burst. The first round is document plus question (cold start), and the second round is the same conversation plus answer plus follow-up question, simulating the agent workflow. Output is capped at 500 tokens, temperature 0.6, thinking disabled, streaming output.

Prefill Is a Fixed Cost; Concurrency Does Not Change Total Throughput

The most critical finding is that prefill throughput is independent of user count, staying stable at about 6,800 tok/s. 8 prompts of 128K total 1.02M tokens, taking 151 seconds. Because the simultaneously arriving cold prompts are processed interleaved, all users get their first token at almost the same moment. This means in the extreme 8×128K scenario, everyone waits about 2.5 minutes.

Compute characteristics of Prefill vs. Decode: The prefill phase processes the entire input prompt at once, with large matrix multiplications and high compute intensity, nearly saturating GPU compute, so throughput manifests as a fixed ceiling independent of concurrency—adding users merely interleaves the same batch of tokens in queue, and total throughput is unchanged. The decode phase generates only one token per step, which is VRAM-bandwidth-bound, and throughput is diluted as concurrency rises. This explains why the total prefill time for 8×128K is roughly the total token count divided by 6800 tok/s, and why adding cards cannot linearly shorten the first long-document processing time.

Prefix cache: Retains the KV cache of already-processed prompt prefixes, so subsequent requests that hit the same prefix can skip recomputation. In multi-turn conversations, the system prompt and historical turns naturally repeat, so hit rates can reach over 90%; the prefill cost of the hit portion is completely eliminated, which is exactly the source of "follow-up turns get their first token in just 0.2-2.5 seconds."

Follow-Up Turns Are the Real Killer Feature of This System

Cache hit rates reach 94% to 99%. Even with 8 users × 128K, the first token takes only 0.2 to 2.5 seconds, with per-user writes of 80 to 225 tok/s. Context length has very little effect on write speed—a single user gets 189 tok/s at 128K and 195 tok/s at 32K, a negligible difference.

First-round write speed, on the other hand, depends on whether other users are still prefilling. With 4 users, the user who finishes first starts writing while others' prompts are still being read, and the slowest user drops to 9 to 32 tok/s. With 8 users, everyone finishes reading at almost the same time, then writes together at 70 to 83 tok/s, which is actually more uniform.

Host Prefix Cache Should Be Sized by User Count × Context

In an early run, the author used --prefix-cache-mib 12288, and 8×64K follow-up turns had only a 71% hit rate, with worst-case first token at 37.6 seconds and only 9 tok/s per user. After changing to 32768, the hit rate was 98%, first token 1.5 seconds, and 94 tok/s per user. This difference is quite significant.

He also points out that the example line for the four-card configuration in the README inherited --host-pool-mib 12288 from the dual-card configuration, but with four cards the entire expert plane resides permanently in VRAM, so the pool only needs about 312 MiB per card. Setting it to 12 GiB instead needlessly occupies 48GB of memory, leaving only 3.6GB free. 1024 is sufficient.

Cooling and Power Observations

During prefill, each card sits at the 210W power limit. The longest 8×128K test segment pushed the middle two cards to 91°C hotspot temperature and 92°C VRAM temperature. Considering the four cards are tightly packed in the case, this temperature is high but still within a reasonable range, and the author has clearly done his homework on airflow design.

Significance for Local Inference Choices

The core value of this configuration is that it anchors a realistic baseline: four R9700s under tp4 can economically handle multi-user agent workloads, especially high-frequency short follow-up scenarios. The fixed prefill throughput means first-time long-document processing cannot be linearly accelerated by adding cards, but for users already in the conversation flow, the experience is quite smooth.

The author's motivation for migrating from Strix Halo is also worth noting—sensitive-data customers previously viewed using any AI inference as a huge risk, so local deployment is not a performance choice but a compliance necessity. Under this premise, 6,800 tok/s prefill and 2.5-second cached first tokens are no longer compromises, but a usable production solution.

The full concurrency tests are still ongoing, and the author says the long-context high-concurrency combination benchmark takes two hours to run. For those considering a similar configuration, the prefix cache and host-pool parameter pitfalls he encountered may be the most valuable public information.

One-line Summary

4x R9700 benchmark: prefill steady at 6800 tok/s, cached first token for 8 users at 128K in just 2.5 seconds.

Paragraph Summary

A Reddit user shared a complete benchmark of a self-built 4-GPU Radeon AI PRO R9700 inference server. The system uses an EPYC 7402P platform, 128GB DDR4, four R9700s power-limited to 210W, costing about CAD 16,000, running Ubuntu and ROCm 10.1, deploying the Qwen3.8 MoE model with the Radiance framework under tp4, with all weights resident in VRAM. Tests show prefill total throughput stays stable at about 6800 tok/s, independent of concurrent user count; the worst case is 8 brand-new 128K prompts arriving simultaneously, where all users wait about 2.5 minutes. Cache-hit follow-up turns perform excellently, with first token in just 0.2 to 2.5 seconds at 8 users × 128K, and per-user writes of 80 to 225 tok/s. The author also points out that host prefix cache should be sized by user count × context, and corrects the host-pool-mib inheritance error in the README.