Building a 64GB Local AI Inference Server with 4x Tesla T4 GPUs: A Full Breakdown

One Enthusiast's Journey Building a Local AI Inference Rig

As local large language model deployment gains momentum, more and more tech enthusiasts are building their own inference servers. A Reddit user recently shared a detailed account of his build — a local inference platform powered by four Tesla T4 GPUs for a combined 64GB of VRAM, backed by substantial RAM and storage. This setup embodies the typical approach to home- or studio-grade AI infrastructure: trading enterprise-grade second-hand hardware for high-value inference capability.

The user mentioned a previous post showing the unfinished state of the build, when he was still waiting on the blower fans for the T4s. Now, aside from finishing touches like sleeving the fan cables, the machine is essentially ready to go. There's a very DIY spirit to the whole thing — the system had barely started running before he was already eyeing more cards.

Hardware Breakdown

The hardware choices here are quite representative of this class of build. Key components include:

Compute & Motherboard

• Motherboard: SuperMicro X11SPA-T — a single-socket workstation/server platform
• CPU: Xeon W3225 — an entry-level Xeon workstation processor
• RAM: 768GB DDR4 ECC 2666 — an impressive amount of memory
• GPU: 4× Tesla T4 16GB — 64GB VRAM total

The SuperMicro X11SPA-T is a high-end workstation motherboard built around Intel's Xeon W series processors. It supports single-socket Xeon W-3200 series CPUs and provides up to 7 PCIe 3.0 slots — critical for fitting all four T4s simultaneously. The board natively supports up to 768GB of DDR4 ECC RDIMM across 12 DIMM slots (up to 64GB per slot). The ECC (Error-Correcting Code) memory can automatically correct single-bit errors on the fly, which is particularly valuable for inference services running unattended over long periods. The Xeon W3225 is an 8-core, 16-thread processor. While the core count isn't high, its strong single-core clock speed (3.7GHz base) and direct PCIe lane count deliver stable performance under I/O-intensive workloads — which aligns with the user's real-world finding that a higher-core-count older CPU didn't outperform it in inference tasks.

Storage & Case

• Storage: 4× 1TB Samsung 870 EVO SATA SSDs in RAID
• Case: Fractal Design Torrent mid-tower (with tinted glass side panel)

The Tesla T4 is NVIDIA's Turing-architecture data center inference card. Its 70W TDP and passive cooling design have made it a popular choice in the second-hand market for local inference. Four cards together provide 64GB of VRAM — enough to load fairly large quantized models.

Software Stack: The Classic llama.cpp + OpenWebUI Combo

On the software side, the user chose:

• OS: Ubuntu
• Inference engine: llama.cpp
• Frontend: OpenWebUI
• Custom tooling: A PowerShell-based task scheduling harness

This is one of the most popular stacks in the local deployment community. llama.cpp is well-regarded for its strong support of GGUF quantized models and cross-hardware compatibility, while OpenWebUI delivers a ChatGPT-like conversational experience. Writing a custom PowerShell harness for task scheduling shows extra investment in automation on this user's part.

llama.cpp was started by Georgi Gerganov in 2023, originally to run Meta's LLaMA models in pure CPU mode on Apple Silicon Macs. It quickly grew into a general-purpose inference framework supporting CUDA, ROCm, Metal, and more. Its core strength lies in support for the GGUF (GPT-Generated Unified Format) quantization format — by compressing model weights from FP16/BF16 down to INT8, INT4, or even INT2 precision, VRAM requirements can be reduced to 1/4 to 1/8 of the original, making it possible to run models that would otherwise never fit in VRAM. OpenWebUI (formerly Ollama WebUI) is a self-hosted web frontend offering a chat interface, multi-model switching, RAG document Q&A, and more. It connects to backends like llama.cpp via the OpenAI-compatible API, giving you a near-commercial-quality experience without writing a single line of code.

A Notable Insight: CPU Is Not the Bottleneck

The most practically useful observation in this entire writeup is about the CPU upgrade decision. The user had originally planned to swap out the CPU, but real-world testing showed that the Xeon W3225 actually performed better at inference than the higher-core-count CPU in his previous machine — so he scrapped the upgrade plan.

This is a genuinely instructive finding for the local inference community. When a GPU is handling the bulk of the compute load, CPU core count often isn't the deciding factor. Memory bandwidth, PCIe lanes, and the GPU's own VRAM and compute capacity are the more critical variables. Blindly stacking more CPU cores doesn't necessarily translate into proportional performance gains. That said, this conclusion is based on one user's subjective experience, and actual results will vary depending on model type, quantization level, and batch configuration.

What This Build Represents — and Its Limitations

Looking at the bigger picture, builds like this reflect a maturing local AI deployment ecosystem. Decommissioned data center hardware (like the T4), large-capacity ECC memory, and an increasingly polished open-source inference software stack have collectively lowered the barrier for individuals to own meaningful inference compute.

That said, there are real limitations worth acknowledging. The Tesla T4's per-card compute and memory bandwidth lag behind today's consumer flagship GPUs and newer data center cards. Running multiple cards in parallel introduces communication overhead. Passive-cooling cards depend on external blower fans, requiring extra thought around noise and thermal layout. For workloads demanding maximum throughput, this platform is better suited to learning, experimentation, and small-to-medium-scale inference than production-grade high-concurrency serving.

But as the user's closing line — "now i want more cards 👀" — makes clear, the appeal of local AI hardware lies precisely in this kind of open-ended tinkering: get it running, then keep iterating.

The Tesla T4 is based on NVIDIA's Turing architecture (launched in 2018), featuring 2,560 CUDA cores and 320 Tensor cores, with FP16 throughput of roughly 65 TFLOPS and INT8 throughput up to 130 TOPS. The passive cooling design means the card has no onboard fan — heat dissipation relies entirely on chassis airflow or external blower fans. That's what the "blowers" mentioned in the build refer to: typically 80mm or 40mm high-speed turbine fans that can exceed 55dB, which may require soundproofing or a dedicated machine room in a home environment. The T4's popularity on the second-hand market stems from its extremely low TDP (70W) and relatively affordable pricing (usually $200–$400 per card). However, its memory bandwidth of only 300GB/s pales compared to the RTX 4090's 1,008GB/s — and memory bandwidth is the key factor governing LLM token generation speed. This is the fundamental reason the platform is better suited to experimentation than high-throughput production workloads.