768GB VRAM for Less Than a Single RTX 6000: Building a Local LLM Rig with Mining Cards

A hardware enthusiast well-known in the modding community recently shared his latest build: 12 CMP170HX mining cards combined to deliver 768GB of VRAM, at a total cost lower than a single RTX 6000 Pro. The project has reignited the classic debate over whether running large models locally is actually worth it.

Stacking Mining Cards to Hit 768GB VRAM

The centerpiece of this build is 12 CMP170HX cards. This is NVIDIA's compute card designed for cryptocurrency mining, based on the GA100 architecture and equipped with HBM2e memory — available at relatively affordable prices on the second-hand market. By running the cards in parallel, the builder accumulated a total VRAM pool of 768GB from individual 64GB cards.

The cost comparison is striking. The RTX 6000 Pro is a professional workstation GPU with a steep price tag — and far less VRAM than this multi-card setup. The builder emphasizes that the entire build came in "under the price of one RTX 6000 Pro," yet delivers several times the VRAM of a single card — which is the critical bottleneck when running very large models locally.

To push memory capacity even further, the builder used a fiber optic connection to link this machine with a second system in an RPC (Remote Procedure Call) cluster, allowing resources from both machines to be pooled when larger VRAM is needed.

CMP (Cryptocurrency Mining Processor) is NVIDIA's product line specifically designed for the mining market, with video output deliberately disabled and certain graphics features stripped out to differentiate it from consumer GPUs. The CMP170HX is based on the same GA100 die as the A100, equipped with HBM2e (High Bandwidth Memory 2e), offering a single-card bandwidth of up to 1.5TB/s — far exceeding GDDR6X solutions. HBM2e stacks multiple DRAM dies vertically and connects them directly to the GPU through a silicon interposer, delivering high capacity alongside extremely high bandwidth density — precisely what large-model inference needs for batched matrix operations. Since GA100 originally powered the data center A100, the CMP170HX has brought this chip into the consumer market at relatively low second-hand prices, making it a go-to "bargain find" for budget-conscious local deployment enthusiasts.

What Models Can It Run?

The builder listed the models actually running on this system, covering most of today's mainstream open-source LLM families: GLM, the DeepSeek^Ad Flash series, the Qwen3 series, KimiK, and MiniMax. For inference frameworks, he uses vLLM and llama.cpp.

His reasoning for needing all that VRAM is straightforward: a single RTX 6000 or an M3 Mac Studio simply can't load models at this scale. VRAM capacity determines how large a set of model weights you can fit in memory, and stacking multiple cards is the cost-effective path around single-card limits.

He also mentioned that since building this setup, he's posted much less frequently — because "chatting with my own machine is more interesting than posting" — a tongue-in-cheek remark that hints at the unique experience of running a local LLM: low latency and complete privacy.

vLLM is a high-throughput LLM inference framework developed at UC Berkeley. Its core innovation is PagedAttention — drawing inspiration from OS virtual memory paging to split KV Cache (key-value cache) into fixed-size blocks for dynamic allocation, significantly reducing VRAM fragmentation and enabling larger concurrent batches on the same hardware. llama.cpp is a lightweight inference engine written in pure C/C++, originally optimized for CPU inference but later expanded to support CUDA, Metal, and other backends. It's known for minimal dependencies and flexible quantization support. The two frameworks each have their strengths: vLLM is better suited for high-throughput service deployments, while llama.cpp offers more flexibility in resource-constrained environments or when fine-grained quantization control is needed. On non-standard configurations like multi-GPU mining cards, llama.cpp is typically easier to adapt than vLLM.

Local Deployment vs. API: The Classic Debate Resurfaces

This kind of post almost inevitably sparks a familiar divide: when cloud APIs are readily available, why spend serious money stacking hardware?

The builder preemptively addressed the three most common objections:

• APIs are cheaper: Pay-as-you-go cloud services have lower upfront costs
• Long payback period: Some commenters joked it would take "52 light-years to break even"
• Noise and electricity: Power consumption and cooling under full load are real ongoing costs

His response to all three is a flat "NOT" — he simply doesn't buy these objections. From a practical standpoint, local deployment does offer things no API can replicate: complete data privacy, freedom from service provider rate limits and price changes, and the ability to freely experiment with models and parameters. That said, the API camp has valid points too — for low-frequency, lightweight use cases, cloud inference usually offers better cost efficiency.

There's no universal right answer here. It comes down to how intensively you use it, how much you care about data sovereignty, and how much you enjoy tinkering.

Catching the Compute Opportunity Window

Toward the end of his post, the builder offers a perspective that sounds almost like investment advice: compute demand will keep rising for the foreseeable future, the second-hand compute card market occasionally throws up great deals, and the key is to stay alert and move decisively when the right opportunity appears.

This view has merit. As open-source models continue closing the gap with closed-source ones, the motivation to run models locally keeps growing — and the wave of mining cards hitting the second-hand market after the crypto mining bust has opened a low-cost entry window for budget-conscious enthusiasts. Cards like the CMP170HX, originally purpose-built for mining, have now found a second life in inference workloads.

That said, mining card builds have clear drawbacks worth noting: CMP series cards have limited video output, reduced driver support, and constrained PCIe bandwidth. Getting multiple cards to work together reliably requires real hands-on skills in software configuration and stability tuning — this is far from plug-and-play. For general users, the barrier to entry remains high.

The "mining bust" primarily refers to the aftermath of Ethereum's 2022 transition from Proof of Work (PoW) to Proof of Stake (PoS) — known as "The Merge" — which eliminated the main use case for GPU mining and flooded the second-hand market with both dedicated mining cards and consumer GPUs, sending prices sharply lower. Cards like the CMP170HX gained their cost-effectiveness window in this context. However, the window isn't permanent. As AI inference demand continues to grow, second-hand compute card prices have already started recovering, and the supply of quality hardware is being consumed. Finding bargain deals will only get harder over time.

Final Thoughts

This 768GB VRAM build is a compelling example of what hardware enthusiasts can achieve under budget constraints. It's not a blueprint everyone should follow, but it does point to a broader trend: as open-source LLMs proliferate, the options for running local compute are becoming increasingly diverse. For users who enjoy tinkering and care about data ownership, stacking second-hand compute cards for VRAM is a path worth exploring.