Qwen3 27B on a Single AMD R9700 with Performance Doubled: 153 tok/s Decode Benchmark Breakdown

New Milestone for Single-Card AMD Local Inference

In the local LLM inference space, AMD GPUs have long been seen as the "second-best" option outside the CUDA ecosystem. However, benchmark results recently shared by a developer on Reddit represent a meaningful performance breakthrough for users running a single AMD Radeon R9700. After targeted optimizations, this setup achieved across-the-board performance doubling when running the Qwen3 27B NVFP4 quantized model.

According to the original author, a large number of users in community comment sections and on Discord had been asking about single R9700 (1xR9700) performance, so he took the time to specifically optimize for that configuration. The end result was "performance doubling across all metrics." All tests were conducted using the Qwen3 27B NVFP4 quantized version released by Unsloth.

Decode Performance: Notable Differences Across Task Types

Looking at the decode phase benchmark data, token generation speeds vary quite significantly across different task types. On JSON tasks — which best showcase the advantages of structured output — the median decode speed reached 153.1 tok/s, the peak figure cited in the title. Math tasks came in at 140.0 tok/s, summarization at 141.7 tok/s, and file_edit at 138.0 tok/s, all in the high range.

By contrast, tasks leaning toward natural language generation ran slower: chat at 67.1 tok/s and prose writing at 69.2 tok/s. Code and reasoning landed in the middle at 120.5 and 123.9 tok/s respectively. This divergence illustrates that token generation speed is not a fixed value — it is closely tied to the structural characteristics and token distribution of the output. Highly structured, predictable outputs tend to achieve higher throughput.

Also worth noting: the update latency (update p50 ms) across most tasks hovered steadily around 42ms, while reasoning and summarization came in even lower at around 34ms, indicating strong response consistency.

Prefill and Concurrency: Where True Throughput Potential Shows

For local inference experiences, prefill speed often matters more than raw decode speed when it comes to real-world feel in long-context scenarios. The test data shows the R9700's prefill throughput is quite robust:

• 2,000 token depth: 3,552 tok/s
• 8,000 token depth: 3,536 tok/s
• 16,000 token depth: 3,619 tok/s (peak)
• 32,000 token depth: 3,437 tok/s
• 64,000 token depth: 3,192 tok/s

Even as context depth increases to 64K, prefill speed only drops from the peak 3,619 tok/s to 3,192 tok/s — a degradation of around 12%. This means the setup won't experience severe performance collapse when handling long documents or lengthy conversation histories.

Concurrency tests reveal its potential as a lightweight inference server. A single request achieves 120 tok/s; 2 concurrent requests bump that to 215 tok/s; 4 concurrent hits 322 tok/s; and 8 concurrent requests push total throughput to 471 tok/s. While concurrency scaling isn't linear (8 concurrent requests don't reach 8× the single-request speed), total throughput grows steadily with concurrency — more than sufficient for personal or small-team self-hosted inference services.

Background: Prefill vs. Decode Prefill and decode are two fundamentally different phases of LLM inference with distinct computational characteristics. The prefill phase processes all input tokens from the user in one pass — it's compute-bound, making full use of the GPU's parallel matrix multiplication capabilities, which is why throughput numbers are typically far higher than during decode. The decode phase autoregressively generates output one token at a time; each step only processes a single new token, making it memory-bandwidth-bound — GPU compute sits largely idle, and the bottleneck is how fast memory bandwidth can "feed" model weights to the compute units. This is why prefill speed in this article reaches 3,600 tok/s while decode speed hovers around 150 tok/s. Concurrent requests improve total decode throughput precisely because multiple requests can be batched together, utilizing otherwise idle compute and spreading out the memory bandwidth bottleneck.

The Role of NVFP4 Quantization

This test used the NVFP4 quantized version of the Qwen3 27B model. NVFP4, as a 4-bit floating-point quantization scheme, dramatically reduces VRAM usage while preserving model accuracy as much as possible — this is the key prerequisite enabling a 27B-class model to run smoothly on a single consumer-grade or workstation-grade AMD GPU.

The author also updated two code repositories to make it accessible across platforms — one hosted on Codeberg (radiance-vllm-mxfp4) and another mirrored to GitHub (vllm-mxfp4) at users' request. The repository names indicate that these optimizations center on the vLLM inference framework combined with MXFP4/NVFP4 quantization paths.

Background: NVFP4 vs. MXFP4 NVFP4 (NVIDIA Float Point 4-bit) is a 4-bit floating-point format that, unlike traditional INT4 integer quantization, retains an exponent field — meaning it incurs less precision loss when representing weight distributions with large dynamic ranges. MXFP4 is part of the MX (Microscaling) standard co-developed by Microsoft, Intel, and others; it's a grouped floating-point quantization spec where each group of weights shares a single scale factor, striking a solid balance between compression ratio and precision. While they differ in naming and implementation details, both fall under the "4-bit floating-point quantization" umbrella. The core idea is to compress model weights stored in FP16 or BF16 down to 4 bits, reducing VRAM usage by roughly 4×, enabling a 27B-parameter model to fit within the 16–24 GB VRAM of a consumer or workstation GPU. The trade-off is an additional dequantization step required at inference time — and optimizing the compute kernels for this step is precisely the core work in the vLLM framework changes described here.

What This Means for the AMD Local Inference Ecosystem

Community-driven benchmarks and optimizations like this are progressively filling the gaps in AMD GPU support for local LLM deployment. NVIDIA has long dominated the space through CUDA and a mature inference framework ecosystem, leaving AMD users often stuck with a "can run, but can't run fast" situation. A single R9700 hitting a 153 tok/s decode peak and 3,600+ tok/s prefill speed demonstrates that, with the right quantization format and framework-level optimization, AMD hardware's inference potential is being steadily unlocked.

It's worth noting that these numbers come from a single developer's report and have not yet been validated by large-scale third-party reproduction. Actual performance may vary depending on driver version, system configuration, and model specifics. For users considering AMD GPUs for local inference, this data serves as a valuable reference — but testing in your own environment is still recommended.

For users who already own a single R9700, the "performance doubling" from this optimization is undeniably a welcome gift — the same hardware, improved throughput purely through software-layer optimization.

Background: AMD's Core Challenge in LLM Inference The core challenge AMD GPUs face in LLM inference isn't the hardware performance itself — it's the maturity gap in the software ecosystem. NVIDIA's CUDA platform has over a decade of accumulated development, with highly optimized inference kernel libraries (cuBLAS, cuDNN, TensorRT) and native support from mainstream frameworks like vLLM and llama.cpp. AMD's ROCm, while a credible counterpart, still lags in kernel optimization coverage, community contributor scale, and framework compatibility — resulting in lower practical inference efficiency for equivalent hardware specs. This optimization is a prime example of that "software catching up to hardware" process: writing or tuning dedicated compute kernels for the MXFP4 quantization format on the ROCm + vLLM path. Similar community-driven optimizations have appeared multiple times in recent years, gradually narrowing the practical usability gap between AMD and NVIDIA, and underscoring the importance of an open ecosystem for non-mainstream hardware platforms.