Building a 128GB VRAM AI Inference Server for $3,000: A Real-World Breakdown

Building a Local LLM Inference Platform with Second-Hand Server Hardware

With cloud GPU rental prices remaining stubbornly high, more and more AI enthusiasts are turning to self-built local inference servers. Recently, a Reddit user shared their complete build: for roughly $3,000, they assembled an inference machine equipped with 128GB of VRAM and 256GB of system RAM, purpose-built for running local large language models. For anyone who wants to deploy open-source models at home without being perpetually billed by cloud services, this build offers a genuinely useful reference.

Worth noting: the builder initially tried a Lenovo P620 workstation. While they felt the P620 offered decent value overall, Lenovo's extensive use of proprietary hardware became a constant frustration, ultimately leading them to return it and go the DIY route. This reflects a classic tradeoff between branded systems and custom builds — convenience and flexibility rarely coexist.

Hardware List and Cost Breakdown

The heart of this build is four AMD Radeon Pro V620 GPUs, totaling around $1,400 and providing 128GB of combined VRAM. This is the key to the whole setup — large VRAM capacity means being able to load models with more parameters and longer context windows, and the V620's relatively friendly second-hand pricing makes it a compelling choice for budget-conscious VRAM stacking.

Here's the full parts list:

• 4× V620 GPUs: $1,400 (128GB VRAM)
• 256GB DDR4 RDIMM 2666: $610
• Huanan Gold D12D motherboard: $410
• EPYC 7452 CPU: $170
• ASRock 1600W PSU: $220
• Samsung 970 EVO 1TB SSD: Already owned
• Case / fans / miscellaneous: ~$200

Pairing an EPYC 7452 with a server motherboard and RDIMM memory is classic "enterprise salvage" territory — getting ample PCIe lanes and memory capacity at low cost, both of which are critical for multi-GPU parallel inference. All told, excluding the SSD already on hand, the total investment comes in just over $3,000.

AMD Radeon Pro V620 is AMD's professional workstation GPU, with 32GB of GDDR6 VRAM per card — 128GB across four cards. Unlike consumer RX-series cards, the V620 uses a full PCIe x16 interface and passive cooling, making it well-suited for dense multi-card installations inside a chassis. For AI inference, the V620 runs PyTorch and vLLM via ROCm (AMD's GPU compute platform, analogous to NVIDIA's CUDA). ROCm's ecosystem maturity and driver stability have historically been AMD's weak points, but support for mainstream inference frameworks has improved significantly in recent years, making V620 a popular choice for second-hand large-VRAM builds.

EPYC 7452 is an AMD second-generation EPYC (Rome architecture) processor with 32 cores and 64 threads, supporting 8-channel DDR4 memory and 128 PCIe 4.0 lanes. Sufficient PCIe lanes are foundational for any multi-GPU build — four V620s theoretically require 4×16=64 PCIe lanes, which the EPYC platform handles with ease. Consumer platforms (such as Intel Z790 or AMD X670) often become a bottleneck for multi-GPU deployments due to their limited lane counts.

Power Consumption and Performance

A system of this scale draws serious power. According to the builder's measurements, the prefill stage hits 700–900 watts, while the decode stage runs at 500–600 watts. Long-term operation demands careful consideration of electricity costs and thermal management — this isn't something most home environments can absorb without planning.

On performance, the builder was initially disappointed — speeds running Qwen3 27B series models fell short of expectations. But switching to Qwen3-Next-Flash (Autoround W4A16 quantization) changed the picture dramatically:

• Prefill speed: ~1.3k tokens/s
• Generation speed: 70 tokens/s (code) / 60 tokens/s (prose)
• Supports 128k+ long context
• Uses MTP-2, running on a fork of vLLM

These numbers illustrate how much the hardware's potential depends on model architecture and inference framework compatibility. The same machine can deliver wildly different experiences depending on what you run. The builder ultimately expressed satisfaction with the build and is looking forward to further gains from future inference optimizations.

W4A16 quantization is a mixed-precision quantization scheme where W4 means model weights are stored as 4-bit integers, while A16 means activations remain at 16-bit floating point. Compared to full precision (FP16/BF16), W4A16 can compress model VRAM usage to roughly one-quarter, making large models that would otherwise not fit into VRAM feasible — and because activations stay at high precision, accuracy loss is far smaller than with fully quantized approaches. Autoround is an open-source weight quantization algorithm from Huawei Noah's Ark Lab that uses an adaptive rounding strategy to preserve model quality at low bit-widths, and is among the top-performing methods for 4-bit quantization today.

MTP (Multi-Token Prediction) is a speculative decoding acceleration technique where the model predicts multiple subsequent tokens in each decoding step, then filters correct results through a verification pass — boosting effective throughput without changing output quality. MTP-2 means two draft tokens are predicted per step. This is a natively supported feature of the Qwen3 series, but requires corresponding support from the inference framework to activate.

Lessons for Local Deployment Enthusiasts

This build highlights several realities of local LLM deployment that are easy to overlook. Hardware stacking is only the first step — the quantization approach (such as W4A16), inference framework choice (a vLLM fork), and the model's own architectural efficiency collectively determine the final usability. The builder's journey from disappointment with Qwen3-27B speeds to satisfaction with Qwen3-Next-Flash is a vivid illustration of this process.

At the same time, assembling a large-VRAM platform from multiple consumer or professional second-hand GPUs — while impressive in total VRAM — means confronting a host of engineering challenges: driver compatibility, multi-GPU communication, power delivery, and thermal management. AMD cards like the V620 have a less mature software ecosystem than NVIDIA, and getting vLLM running smoothly at solid speeds requires real technical effort.

For budget-conscious users who want a fully autonomous inference environment, this $3,000 build provides a viable reference template. But it's best suited for technically inclined tinkerers who are willing to tune deeply and can accept high power draw — not for users looking for a plug-and-play experience.