Two DGX Sparks vs. M5 Ultra: Which Wins for Local AI?

Choosing local AI hardware has become a genuine dilemma for many developers and AI enthusiasts. Shortly after Bilibili creator 小蛋糕 picked up two DGX Sparks, Apple announced the M5 Ultra — so he ran a series of real-world tests and analyses to answer one core question: for running local LLMs, should you buy dual Sparks or an M5 Ultra? The answer might surprise you — the M5 Ultra may not be nearly as dominant as people think.

Compute & Bandwidth: Reading the M5 Ultra's Spec Sheet

Comparing memory bandwidth and compute per dollar, the M5 Ultra's performance is roughly equivalent to two M5 Maxes — exactly what was expected before launch. In BF16 compute, however, it's only about half a Spark. Interestingly, once you move to FP4 precision, the gap narrows considerably, because the M5 Ultra's FP8 and FP4 floating-point throughput matches its BF16 throughput.

For reference, two Sparks are roughly equivalent to half an RTX Pro 6000, which has climbed in price from $10,000 to $15,000. In pure price-to-performance terms, dual Sparks are far from uncompetitive.

Many people see M5 Ultra's memory bandwidth lead and immediately conclude it's the stronger all-around performer. But the creator's real-world analysis shows it's far more nuanced than that.

Long-Context Degradation: Spark's Hidden Advantage

The real differentiator is performance degradation under long-context workloads.

Real-world tests show that dual DGX Sparks maintain a stable input speed of roughly 2,300 tokens/sec from 0 to 64K context, only declining slowly beyond that. The M3 Ultra, by contrast, starts degrading around 16K, and by 500K its incremental speed has dropped to one-third or one-quarter of peak. At the same context length, the Spark still holds around half its peak speed.

There's an easy-to-miss statistical trap here: input speed can be measured as either "incremental" (the speed at the current position) or "average" (averaged over the entire context). Online benchmarks rarely specify which they're using, and the average metric always looks better. For agentic workflows, incremental throughput reflects the real user experience far more accurately.

Extrapolating from the M5 Ultra's roughly 3–4× compute advantage over M3, its long-context input speed may still only be about half that of dual Sparks. Given that long contexts are becoming increasingly common in agent workflows — especially the "long-context cold-start" problem in coding scenarios, where waiting on a Mac can be genuinely painful — this deserves serious attention.

Why does long-context performance degrade? The root cause lies in the Transformer attention mechanism: computational complexity grows with sequence length, and KV Cache (key-value cache) memory footprint expands linearly with context. KV Cache must be read from memory in real time, making memory bandwidth one of the core bottlenecks in long-context inference. Once the KV Cache exceeds the chip's cache capacity, each decoding step requires frequent large data fetches from main memory, dramatically increasing bandwidth pressure and dropping speed. Some hardware also sees significant efficiency drops in attention computation for very long sequences — the M5 microarchitecture's lower efficiency on long-context attention and indexing, mentioned in the analysis, is a direct example of this. The distinction between incremental and average statistics becomes especially important here: average stats mask the severe speed cliff at the tail end, whereas the actual wait time per new token in an agent loop is what users truly feel.

Speculative Decoding: NVIDIA's Software Moat

Speculative decoding is where Spark really pulls ahead. On Sparks, speculative decoding delivers a full 2× speedup — and this is precisely why the once-overlooked Spark has become so desirable.

The creator offers a sharp analogy: NVIDIA is like a "low-interest loan" to the entire AI industry — while Apple is still figuring out how to trade memory for bandwidth and compute, the entire ecosystem is busy helping NVIDIA figure out how to use spare compute to compensate for bandwidth limitations. Speculative decoding is the perfect example: on NVIDIA GPUs, abundant spare compute can dramatically compensate for memory bandwidth constraints.

On Apple Silicon, the observed speculative decoding gains on M5 are only 4–40%, and M3 Ultra shows almost none — in code and text generation, enabling or disabling it makes little difference. To be fair, this may be a software limitation. A project called MTP LX has already achieved 2× decoding speedup on M5 Max for certain models. If M5 Ultra can achieve that 2× gain, it would surpass Spark in coding scenarios; if not, the advantage is much less clear.

In testing, dual Sparks output a stable 40–45 tokens/sec, maintaining that range even under long-context loads. If M5 Ultra's speculative decoding gains remain limited, the gap versus Spark in long-context scenarios isn't actually that large.

What is speculative decoding? It's a technique that accelerates large model inference by introducing a small "draft model." The draft model rapidly generates several candidate tokens, which the large target model then verifies in parallel. Since verification can be batched and parallelized while the draft model is fast, this significantly reduces the number of serial inference steps the large model must perform — boosting token generation speed without sacrificing output quality. The key prerequisite is "compute surplus" — GPU compute units sitting idle due to memory bandwidth bottlenecks, with spare capacity available to run the draft model and verification in parallel. NVIDIA GPUs' architecture, where compute far exceeds memory bandwidth, naturally suits this optimization. Apple Silicon is designed to balance compute and bandwidth more evenly, leaving little spare compute, which is why speculative decoding yields far less benefit on Apple Silicon than on CUDA platforms.

Multi-Node Interconnect: Where NVIDIA's "Precision Cuts" Land

Scaling across multiple nodes is an unavoidable topic. Interconnect involves two dimensions: latency (which affects the decode phase, since tensor parallelism involves many small message operations) and bandwidth (which affects long-context prefill and large-batch inference).

Tests found that the CX7 on Spark claims 1.5 µs latency, but because RDMA must route through the CPU before reaching the GPU, actual latency is around 20 µs. The creator noted that "Jensen's precision cuts are remarkably sharp" — this result is poor compared to NVLink, but still excellent compared to Thunderbolt 5 or PCIe.

For a personal two-node setup, latency and bandwidth overhead amount to only about 5%. But if output exceeds 100 tokens/sec, latency overhead exceeds 10%; and for large-scale expansion to compete with cloud providers, it's simply not feasible. On the Apple side, Thunderbolt 5-based RDMA has just arrived. The physical latency looks higher, but accounting for the full signal path it may be comparable to Spark's — it's just bandwidth-constrained, which will hurt large-batch and long-prefill workloads more.

What is RDMA? Remote Direct Memory Access allows computers on a network to directly read and write each other's memory, bypassing the OS kernel — commonly seen in data center InfiniBand and RoCE networks. The ConnectX-7 (CX7) NIC in DGX Spark is based on InfiniBand/RDMA standards, rated at 1.5 µs latency. However, as noted, Spark's RDMA path must "route through the CPU before reaching the GPU," pushing end-to-end latency to around 20 µs — significantly worse than NVLink's direct GPU-to-GPU connection (typically sub-microsecond). NVLink is NVIDIA's proprietary high-speed GPU interconnect, with bandwidth and latency far superior to any PCIe or network-based solution — but it's reserved for high-end products like the DGX H-series. Spark, as a personal edge device, doesn't include it. That's exactly what the creator means by "Jensen's precision cuts" — perfectly capping the ceiling on multi-node scaling.

Software Ecosystem: Where M5 Could Still Improve

One potential wildcard for the M5 Ultra is software ecosystem maturity. NVIDIA's stack — vLLM, SGLang, and others — is extremely mature, and if Apple's ecosystem catches up, the M5's theoretical ceiling is worth watching.

The bad news first: M5's microarchitecture still appears to lag behind NVIDIA Blackwell by a generation, with notably lower efficiency on long-context attention and indexing operations — which may explain its faster performance degradation.

There are two promising technical avenues on the good side. First, int8 matrix cores: M5's int8 throughput is 2× its BF16, and FP4 can be represented cleanly in int8. GitHub projects already attempt to use M5's int8 cores for inference and report 1.6× improvements — though int8 activation quantization affects model quality, and no mainstream inference engine currently does this, so the technology isn't mature yet. Second, Apple's ANE (Neural Engine) was recently reverse-engineered, with one report of a 50% prefill improvement on M3. If both avenues pan out, M5 generation input speed could roughly double over current numbers, significantly closing the gap with Spark.

Thermals & Memory: Details That Matter

Thermal performance differs notably. The Spark isn't as loud as you might expect at full load — noise levels are comparable to an M3 Ultra — but internal temperatures spike and drop very quickly. Under sustained load for two or three minutes, even the better-built ASUS version of the Spark pushes the CPU above 90°C. Under the same load, an M3 Ultra stays around 60°C, hitting only about 72°C even under prolonged full load. If you want a hassle-free experience, Apple's thermal stability is in a completely different league — the Spark may require thermal modifications.

On memory utilization, multi-node Spark has a small disadvantage: each machine must run the OS and the VLM server itself, so the actual memory available for models and KV Cache is somewhat less than Apple's unified memory figure. In extreme cases running very large models, the available KV Cache on multi-node Sparks can be a bit tighter.

The Real Value of Local AI — and the Final Choice

If you're purely counting electricity costs, local AI can't compete — machines like the B200 use two orders of magnitude less power per token than the GB10. The real value of local AI lies in privacy (handling sensitive data), speed (a good machine can outpace any cloud API, invaluable for startups and urgent R&D), support for niche models, and reliability (cloud providers vary wildly in quality; a well-configured local setup won't get silently nerfed, and lets you test reliability at very long context lengths).

The creator's final decision: don't buy the M5 Ultra, keep the Sparks. Beyond performance, the bigger factor is that he's running AI experiments where the CUDA ecosystem's ready-to-use libraries offer far less friction than Apple's ecosystem, and his experiments are more compute-intensive. That said, if you prioritize video transcoding or want a home server hub, Mac's advantages there remain substantial.

The takeaway: dual Sparks in absolute performance terms are not necessarily weaker than M5 Ultra — not just on price-to-performance, but in raw performance — and it depends heavily on whether your workload is prefill-heavy or decode-heavy. As inference harnesses improve and cache hit rates rise, the decode share will grow, but the cold-start wait from switching between many conversations in personal use remains a key experience variable. What will truly determine M5 Ultra's value is its actual performance curve under long contexts — if it can sustain 900 tokens/sec at 256K, usability improves dramatically. As for the creator himself, he's waiting for M7.