From One RTX 3090 to Twenty DGX Sparks: One Local AI Enthusiast's Hardware Evolution

An Obsession That Started with LLaMA

When the first-generation LLaMA 33B was released, a Reddit user was captivated by what he described as "near-magical intelligence." For him, the appeal wasn't raw performance — it was ownership. He wanted a local AI that was entirely his own, one that no one could ever take away. So he bought his first RTX 3090 for his home PC, and set off down a remarkably extreme path of local deployment.

This story is worth telling not only because it shows how a consumer-grade enthusiast can run cutting-edge open-source models with trillion-scale parameters at home, but because it honestly captures the engineering bottlenecks that emerge when scaling local LLM inference: from VRAM and RAM, to network bandwidth, and finally to the most elemental physical constraint of all — the circuit breaker in his house.

Stacking VRAM and RAM, Layer by Layer

LLaMA 65B left him awestruck — "it seemed to possess all the knowledge in the world." So determined was he not to lose those model weights that he made two copies: one local, one stored on a Synology NAS in a RAID array. He also added a second 3090 to run it.

For the next year, he used LLaMA and Qwen for everyday coding tasks and was genuinely satisfied. The real turning point came with DeepSeek^Ad 671B MoE — "a frontier-class model you could run at home." To handle it, he upgraded to a Threadripper platform with 512GB of DDR4 RAM, offloading expert layers to system memory to run the 671B at around 8 tokens/s, or Qwen 235B at 10–12 tokens/s when speed was the priority. He actually used these models for real coding work through OpenWebUI.

This phase illustrated the canonical strategy for running MoE models locally: use large-capacity RAM to absorb some of the expert weights, trading speed for the ability to run massive models at all.

When Agentic Coding Hits a Speed Wall

As agentic coding took off, the sluggishness of pure memory offloading became unbearable. He identified a critical pain point: at 100k context length, generation speed dropped by half, and the prefill phase made the experience "beautiful but agonizing."

Performance degradation at long context is a reality every local inference user eventually runs into. His response was dramatic: 16 RTX 3090s distributed across multiple nodes built on Lenovo P620 workstations, interconnected over a 100Gbit network. This setup could run MiniMax M2, Qwen 235B, and even Qwen 397B — "as good as anyone could hope for." He completed a paid client project using the 397B model through OpenCode.

The Performance Gap Between Prefill and Decode

LLM inference has two distinct phases: prefill and decode. The prefill phase processes all input tokens at once, and its computational cost scales roughly quadratically with context length — meaning at 100k context, users may wait tens of seconds before seeing the first output token. The decode phase generates tokens one by one autoregressively; its speed is bottlenecked by memory bandwidth rather than compute. In memory-offloading setups, each generated token requires shuttling the corresponding KV cache and weights from CPU memory to the GPU, making round-trip bandwidth the hard limit. At long contexts, the KV cache itself grows linearly with length, further squeezing available VRAM and causing generation speed to drop by 50% or more compared to short inputs. This is the engineering root of what the author describes as "speed halving at 100k context."

The First Real Bottleneck: The House's Circuit Breaker

Still, the largest models remained out of reach — and what ultimately stopped the expansion wasn't an algorithm or a budget, but the house's wiring: "Whenever this setup ran alongside an electric oven, the fuse would blow." Heat and stability were problems too.

This is the most vivid and honest moment in the entire story. Sixteen RTX 3090s at full inference load draw roughly 6kW — far beyond what a consumer residential circuit can handle. Sometimes the limits of local AI aren't technical at all; they're in your electrical panel.

Switching to DGX Spark: A Dramatic Drop in Power and Noise

The turning point came with NVIDIA's GB10, marketed as DGX Spark. After reading that these units could be interconnected, he bought four ASUS GB10s (the officially supported maximum at the time was three) and ran his setup anyway. The results were stunning: the 397B model ran at 30 tokens/s on just 400W — compared to 50–60 tokens/s at 6kW with the old rig, the new setup was rock-solid and nearly silent.

"397B at 30 t/s on 400W, versus 50-60 t/s at 6kW, rock solid and almost silent. A dream come true."

Dropping from 6kW to 400W, from noisy to near-silent — this leap in energy efficiency is precisely what a unified memory architecture (like GB10's Grace-Blackwell Superchip) brings to local LLM inference. He went on to complete two more client projects using this system.

Unified Memory Architecture (UMA) vs. Traditional Multi-GPU Setups

DGX Spark (GB10) uses the Grace-Blackwell Superchip, whose key advantage is a Unified Memory Architecture: the CPU and GPU share a single physical memory pool (up to 288GB) with 900GB/s of bandwidth, with no need to move data across a PCIe bus. In traditional multi-GPU setups, model weights are spread across GPU VRAM and host DDR, and PCIe bandwidth (roughly 64GB/s for Gen4 x16) often becomes the hard bottleneck in memory-offloading scenarios. GB10's on-package high-bandwidth memory places all weights at the same fast access distance. This explains how just four GB10 units at 400W can achieve over 60% of the throughput of sixteen RTX 3090s at 6kW — the efficiency gain doesn't come from cheaper chips, but from eliminating cross-bus memory transfer overhead at the architecture level.

Breaking Official Limits: 8-Node and 16-Node Clusters

As smarter models like MiMo 2.5 Pro and Kimi 2.6 appeared, he kept scaling. Unable to find any public documentation for an 8-node cluster, he bought four more GB10s and figured it out himself — allowing the 397B to run in FP8 (rather than INT4) with a 20% speed boost. He published the first known configuration for running MiMo 2.5 Pro and Kimi 2.6 on eight Sparks to the NVIDIA forums.

What followed had an unexpected social dimension: he convinced his brother, who lives a five-minute walk away, to buy his own 8x GB10 setup. When the 2.8-trillion-parameter Kimi K3 was released, the two brothers merged their clusters — using two independent 8-node setups day-to-day, then combining them into a single 16-node cluster for the largest models. He published the first working configuration for Kimi K3 on 16 Sparks, and through multiple iterations, improved performance from an unusable 7 tokens/s at 100k context to a quite usable 20 tokens/s at 300k context.

The Role of vLLM and SGLang in Multi-Node Inference

vLLM and SGLang are the leading open-source LLM inference engines, both supporting tensor parallelism and pipeline parallelism to shard a single model across multiple GPUs or nodes. In a multi-node Spark cluster, the inference engine coordinates KV cache synchronization across nodes, partitions attention computation, and handles expert routing for MoE models — with inter-node communication latency directly affecting overall throughput. The author's repeated need to "rebuild vLLM/SGLang images" reflects the reality that edge cases like multi-node setups, ultra-long context, and non-standard node counts often require targeted configuration tuning or even code modifications. Community support for Spark clusters beyond 8 nodes was essentially nonexistent at the time, which is why his published configurations had genuine practical value.

Why Stay Purely Local?

They were planning to add four more Sparks, running a smaller, faster GLM 5.3 Flash around the clock on a dedicated cluster, while the larger cluster rotates between GLM 5.3, MiMo 2.6 Pro, 16x Kimi K3, and Qwen 3.8 (2.4T). The always-on smaller cluster serves a practical purpose: since he's constantly tuning speeds and rebuilding vLLM/SGLang images for the big models, he needs a service that's always available.

Most telling is his philosophical statement: for all work projects and personal vLLM/SGLang work, he uses only locally hosted models and has never paid for a commercial model subscription. The reason isn't cost — it's "strong conviction in the future of local models," free from API availability concerns or rising subscription fees.

The story ends with a touch of humor: his younger brother has now caught "the same local AI virus," with a fresh batch of hardware waiting to be delivered.

What This Case Means for Local AI Enthusiasts

This journey from one RTX 3090 to twenty DGX Sparks is fundamentally a chronicle of shifting bottlenecks in local LLM engineering. It clearly maps out the sequential constraints: single-card VRAM → system RAM (MoE offloading) → long-context performance degradation → multi-card power draw and thermals → the absence of multi-node interconnect solutions.

For anyone hoping to run frontier open-source models at home, the most practical takeaway is this: once you stack more than a dozen GPUs, power and cooling tend to become the ceiling before budget does. And unified-memory, low-power dedicated inference hardware is emerging as the more sustainable path for home-scale frontier model deployment.

Background: How MoE Models Work

MoE (Mixture of Experts) Architecture

MoE is a sparsely-activated model architecture. Take DeepSeek 671B as an example: although the model has 671 billion parameters, each inference pass only activates a small subset of "expert" sub-networks (typically less than 20% of total parameters), so actual compute is far lower than a dense model of equivalent size. This gives MoE models an asymmetric profile at inference time — large storage requirements, but relatively modest compute. It creates a viable space for CPU memory offloading: store less-frequently-used expert weights in large-capacity DDR RAM, swap them into GPU VRAM on demand, trading higher access latency for the ability to run enormous models on consumer hardware. The 512GB DDR4 configuration was specifically chosen to hold these "cold expert" weights.