Distributed LLM Inference on Mac Studio with MLX and RDMA

Running frontier large models locally usually means facing the awkward reality of insufficient VRAM and compute. Apple's latest update to the MLX framework offers another approach: linking multiple Mac Studios via Thunderbolt 5 RDMA drivers, enabling models with nearly a trillion parameters to run smoothly on a desktop.

According to a technical article published by the Apple MLX team, a cluster of four Mac Studios running an 816B-parameter Muse Spark Open achieves decoding speeds exceeding 36 tokens/s, clearly above average human reading speed. This is not a proof of concept—it is a result that can already be reproduced in the latest version of MLX.

Single-Machine Inference: Two Key Metrics for the M5 Ultra

Apple evaluated three models on Mac Studio using mlx_lm.benchmark: the 816B Muse Spark Open (MoE architecture, NVFP4 quantized), the 128B Mistral Medium 3.5 (FP8), and the 27B Qwen 3.8 (BF16). The latter two both fit on a single Mac Studio with 512GB of memory, occupying approximately 158GB and 68GB respectively in a large-context scenario of 65536 tokens.

The two core metrics for measuring inference experience—time to first token (TTFT) and decoding speed—both show significant improvement on the M5 Ultra. TTFT is fundamentally a compute bottleneck, and this is where the M5 Ultra GPU neural accelerator shines, with its dedicated matrix multiplication hardware improving TTFT for Qwen 3.8 and Mistral Medium 3.5 by up to 4.6x. Decoding is a memory bandwidth bottleneck, and the M5 Ultra's unified memory bandwidth has increased from 819 GB/s on the M3 Ultra to 1.2 TB/s, making decoding 1.6x faster accordingly.

Power consumption figures are also worth noting: the M3 Ultra Mac Studio peaks at around 200W during inference, while the M5 Ultra draws about 300W—far below what many imagine for "running large models." This power level means it can be deployed directly in ordinary homes or office environments without dedicated server-room cooling.

Time to first token (TTFT) and decoding speed are two key metrics for measuring large model inference experience, but the hardware bottlenecks behind them are entirely different. TTFT refers to the time from when a user sends a request to when the model emits its first token, primarily determined by the prefill phase—the model must process the entire input context at once, with computation growing nearly linearly with context length, making it a classic compute bottleneck that depends on GPU matrix multiplication throughput. Decoding speed refers to the rate at which each subsequent token is generated; in this phase only one token is processed per step, so computation is minimal, but model weights and KV cache must be moved from memory to compute units, making it a classic memory bandwidth bottleneck. Understanding this clarifies why the M5 Ultra's two improvements come from different hardware: the GPU neural accelerator strengthens compute to shorten TTFT, while 1.2 TB/s unified memory bandwidth speeds up data movement to boost decoding speed.

When the Model Won't Fit: The Division of Labor Between Two Parallel Strategies

Muse Spark Open is the most extreme case in this test. Even quantized to 4-bit (NVFP4), the 816B-parameter model itself occupies about 493GB of memory, and adding intermediate computation overhead during inference, it simply cannot fit on a single machine. It must be split across machines.

MLX offers two paths. Pipeline parallelization splits by depth: the first Mac Studio executes the first half of the network layers and passes the output to the second machine for the second half. The implementation change is minimal, and decoding speed is nearly identical to single-machine operation because layers still execute sequentially. However, in large-context scenarios, pipeline parallelism can actually outperform single-machine—MLX splits the 65536-token context into 8192-token chunks, with the two machines processing adjacent chunks' earlier and later layers, creating temporal overlap that speeds up prefill by up to about 1.8x.

Tensor parallelization splits by width, with each machine holding a portion of every layer, parallelizing computation within layers. This approach can genuinely improve inference speed, but implementation complexity is far higher: every token, every layer requires broadcasting inputs and aggregating outputs, resulting in numerous and fragmented communication operations. Without a low-latency link like Thunderbolt RDMA, tensor parallelism's gains would be consumed by communication overhead.

Apple uses the jaccl communication collective (supported since MLX v0.30.1, based on the Thunderbolt 5 RDMA driver introduced in macOS 26.2) to support both parallelism strategies. The mlx.launch command makes switching parallel strategies a matter of changing a single parameter.

MoE (Mixture of Experts) is a sparsely activated model architecture: the model contains multiple parallel "expert" sub-networks, but each token is routed to only a few of them for computation, with the remaining parameters idle. This allows MoE models to have extremely large total parameter counts (such as 816B) while the actually activated parameter count per step is far smaller, improving model capacity while keeping inference costs manageable. The trade-off is that MoE is more communication-sensitive—when experts are distributed across different devices, tokens need to be routed and gathered, and expert outputs must be normalized by routing weights before merging; these extra operations introduce considerable communication overhead in distributed inference. This is precisely the core reason why Muse Spark Open in the article scales less well than dense models in few-machine scenarios.

Scale Effects: Tensor Parallelism Only Pays Off with More Machines

Tensor parallelism's gains are strongly correlated with machine count. On two M5 Ultra machines, communication overhead offsets part of the compute gains at very large contexts—Muse Spark Open at 65536 tokens with two-machine tensor parallelism is even worse than two-machine pipeline parallelism. But adding to four machines reverses the situation: Muse Spark Open's decoding speed goes from 15.74 tokens/s with two-machine pipeline to 36.73 tokens/s with four-machine tensor, and Qwen 3.8 from 19.44 to 50.89 tokens/s.

The connection method also affects performance. The test used dual Thunderbolt cables for networking, with two cables connecting each pair of machines to double link bandwidth. For Muse Spark Open on four M5 Ultra machines at 65536-token context, dual cables reduce TTFT from about 50.5 seconds to 40.6 seconds—a clear indication of the prefill phase's bandwidth sensitivity.

An interesting observation: dense models (Qwen 3.8, Mistral Medium 3.5) scale better across multiple machines than MoE models. Muse Spark Open's expert outputs require separate normalization, introducing additional prefill communication overhead that is especially pronounced at low machine counts.

NVFP4 and FP8 are two low-precision numerical formats used to compress memory footprint and improve computational efficiency with virtually no loss in model accuracy. FP8 uses 8-bit floating point to represent each weight, directly halving memory compared to traditional FP16/BF16. NVFP4 goes further, using 4-bit floating point (about 16 representable values) combined with group scaling strategies to compensate for the error introduced by extremely low precision, achieving a typical compression ratio of one-quarter of FP16. Taking the 816B-parameter Muse Spark Open as an example, storing it in FP16 would require over 1.6TB of memory, whereas NVFP4 quantization brings it down to about 493GB—making distributed deployment across four Mac Studios possible. The choice of quantization format directly affects how large a model you can run and how many machines you need, making it a key part of local inference solution design.

The Practical Significance of Local Large Models

The real value of this solution is not in benchmarks, but in how it changes the deployment boundaries of large model inference. A single Mac Studio with 512GB of memory can host 128B-class dense models for agentic coding scenarios requiring long contexts; a four-machine network can reach nearly trillion-parameter MoE models. Everything runs in a private environment, with data never leaving the local site.

For teams that need to handle sensitive data while wanting to use frontier models, this "desktop cluster" path deserves serious evaluation. Its barrier is not racks and cooling, but whether you are willing to accept the software stack constraints of the Apple ecosystem.

More technical details and complete benchmark data can be found in the official MLX documentation and MLX distributed documentation.

Background Supplement

RDMA (Remote Direct Memory Access) is a technology that allows one computer to directly read and write another computer's memory, with data movement bypassing both operating systems and CPUs, performed directly by network card hardware. Traditional network communication requires going through the kernel protocol stack, multiple memory copies, and CPU interrupt handling, resulting in high latency and CPU resource usage; RDMA offloads this entire process to hardware, achieving microsecond-level latency and extremely low CPU overhead. The Thunderbolt 5 RDMA driver Apple introduced in macOS 26.2 brings this capability to consumer Macs, making collective communications such as AllReduce and broadcast between multiple Mac Studios fast enough that communication-intensive strategies like tensor parallelism become truly practical—this is the technical cornerstone that makes the entire "desktop cluster" solution viable.

One-line Summary

Apple MLX uses Thunderbolt 5 RDMA to network multiple Mac Studios, enabling local inference of nearly trillion-parameter models.

Paragraph Summary

The Apple MLX team demonstrated a new approach to distributed large model inference by clustering multiple Mac Studios via Thunderbolt 5 RDMA drivers. Four M5 Ultra machines running the 816B-parameter Muse Spark Open achieve decoding speeds over 36 tokens/s. On a single machine, the M5 Ultra improves TTFT by up to 4.6x over the M3 Ultra (thanks to the GPU neural accelerator) and decoding speed by 1.6x (1.2 TB/s memory bandwidth), with peak power consumption of only 200-300W. Cross-machine inference offers two paths: pipeline parallelism, which is simple to implement and speeds up large-context prefill by up to 1.8x; and tensor parallelism, which accelerates significantly with more machines but has high communication overhead and requires low-latency links. Testing also found that dense models scale better across multiple machines than MoE models. This solution makes running frontier models in private environments possible.