4x AMD R9700 Local AI Server for $18K: Real-World Performance for a 10-Person Team

A Local Inference Setup for Small Teams

A developer shared on Reddit a local large model inference server build for a client — approximately $18,000 in total, designed to provide a private AI inference capability for a 10-person startup. This kind of need is becoming increasingly common: teams want large model capabilities without continuously sending code and data to third-party APIs, making local deployment an attractive middle ground.

The highlight of this build is four AMD Radeon R9700 AI Pro GPUs delivering 128GB of combined VRAM, with careful power management keeping total system draw within a manageable budget. For small teams that are cost-conscious but sensitive about data privacy, this is a real-world example worth examining.

Hardware Breakdown

The key specs for the full system:

• CPU: Threadripper 9970X
• RAM: 128GB DDR5 ECC 5600
• GPU: 4× AMD Radeon R9700 AI Pro, 32GB VRAM each, 128GB total VRAM
• PSU: 1600W

One easily overlooked detail is power management. The original post mentions all GPUs were undervolted, with each card kept under 210W. Four cards at full load draw roughly 840W combined — leaving the 1600W PSU with ample headroom once CPU and other components are factored in. This approach is highly practical for multi-GPU local deployments: undervolting reduces electricity costs and thermal pressure, and allows the system to run stably for extended periods on standard circuits and in typical office enclosures — especially useful for small team environments.

Choosing AMD over NVIDIA was largely a cost-efficiency decision. Per gigabyte of VRAM, the total procurement cost for professional cards with 32GB each tends to be lower than comparable NVIDIA options at the same capacity — a key reason the overall build came in at $18,000.

Software Stack and Model Selection

For the inference engine, the author used a fork of Radiance to serve models. The models running on the system include the Qwen3 series (27B and the Next flash variant) as well as DSV4 Flash. KV cache uses BF16 precision throughout, while model weights use MXFP4 quantization.

The logic behind this combination is clear: MXFP4 quantization compresses model size to fit within limited VRAM and improve throughput, while BF16 KV cache preserves quality stability over longer contexts. For a team serving multiple engineers simultaneously, aggregate throughput and concurrency matter more than minimizing latency for a single request — this configuration is clearly optimized for concurrent serving.

What is MXFP4? MXFP4 (Microscaling Float Point 4-bit) is a new quantization format standard co-developed by Nvidia, AMD, Intel, and others — part of the MX-series microscaling floating point formats. Unlike traditional INT4 or NF4 quantization, MXFP4 shares a single scaling factor across a group of elements (typically 32), preserving better numerical range at very low bit widths. Compared to INT8 quantization, MXFP4 can cut model weight storage roughly in half, making it possible to fit larger models into limited VRAM. BF16 (Brain Float 16) is Google's 16-bit floating point format that offers a wider exponent range than FP16, making numerical overflow less likely — it maintains more stable computational precision in KV cache scenarios. Combining the two — MXFP4 for weights to save VRAM, BF16 for KV cache to preserve precision — is the mainstream engineering strategy for balancing throughput and quality in multi-GPU local inference.

Benchmark Results

The author provided fairly detailed benchmark results, with key metrics focused on concurrent sessions, prefill throughput, and decode throughput.

16 Concurrent Sessions · Qwen3 27B (MXFP4)

• Aggregate prefill: 6,300–6,800 tok/s
• Aggregate decode: starts at 80 tok/s, peaks around 900 tok/s
• Per-user context range: 4K–128K

Qwen3 Next Flash

• 48K context per user: prefill 6,880 tok/s, aggregate decode 538.5 tok/s
• 128K context per user: prefill 3,050 tok/s, aggregate decode 365 tok/s

Two patterns emerge from the data. First, as per-user context stretches from 48K to 128K, prefill throughput nearly halves (6,880 → 3,050 tok/s) — consistent with the expected quadratic growth in attention computation with context length. Second, even at 128K ultra-long context, the system maintains hundreds of tok/s in aggregate decode speed, which is more than sufficient for a team's daily code completion, document Q&A, and agent tasks.

Importantly, these are aggregate throughput figures (summed across all concurrent sessions). Individual user experience will be roughly the aggregate value divided by the number of active concurrent users. When all 16 sessions are running simultaneously, per-user experience will decrease proportionally — but for a realistic office scenario where not everyone makes requests at the same moment, this build has substantial headroom.

Understanding Prefill vs. Decode: Prefill and Decode are the two phases of large model inference. Prefill is when the model processes the full input prompt in one pass, computing attention representations for all tokens and populating the KV cache — highly parallel with high throughput, but processing time scales roughly quadratically with context length. Decode is the autoregressive token-by-token generation phase, limited by memory bandwidth rather than compute. This is why ultra-long contexts cause a large drop in prefill throughput (as seen above, 6,880 → 3,050 tok/s from 48K to 128K), while decode speed remains relatively stable. For short-output tasks like code completion, decode speed is the primary determinant of perceived latency; for long-input tasks like document summarization, prefill latency is the more critical factor.

Takeaways for Small Team Local Deployments

The value of this case isn't any single performance record — it's that it demonstrates a replicable engineering balance point: using consumer/prosumer AMD multi-GPU cards + aggressive power management + quantized models to deliver private, multi-model inference for a 10-person team under $20,000.

A few practical lessons worth noting:

• VRAM capacity is the priority. 128GB VRAM enables simultaneously hosting multiple models or supporting ultra-long contexts — the critical threshold for practical local deployment.
• Undervolting is a necessity, not an option, for multi-GPU builds. It directly determines the system's deployability and operating costs.
• Aggregate throughput data must be interpreted in the context of real concurrency patterns — it cannot be treated as a single-user experience metric.
• The AMD ecosystem is now viable for inference workloads. Paired with an appropriate inference engine (like the Radiance fork used here), it delivers solid results.

That said, this is a single-source personal build report — the specific numbers haven't been independently reproduced, and different model versions, quantization strategies, and engine implementations will significantly affect results. For teams considering building their own inference servers, this is best used as a reference for framing your own evaluation, not as a configuration to copy wholesale.