Building a Distributed GPU Cluster from Idle Macs: A Complete Hands-On Guide to LLM Inference

Why Use Idle Macs for LLM Inference?

Most developers' Macs sit idle overnight, yet a single M2 Max Mac Studio can already run Llama-2-7B inference at 8 tokens per second. When you have four Apple Silicon machines in the lab, linking them into a distributed inference cluster turns idle compute into a practical LLM serving capability.

This article is based on a complete technical hands-on guide, breaking down the full workflow from network topology discovery, unified memory optimization, and Ray+Exo orchestration to monitoring and alerting.

Step 1: Inventory Your Hardware and Build the Network Topology

The starting point for building a cluster is enumerating all Apple Silicon devices on the local network. Use arp-scan --localnet | grep -i apple to return MAC addresses and hostnames.

A typical lab setup might include: two M2 Max (38-core GPU, 400 GB/s bandwidth), one M3 Ultra (80-core GPU, 800 GB/s), and one M1 Pro (16-core GPU). You'll need to record each host, GPU core count, and unified memory size into a YAML config file.

Key action: run exo cluster init --config inventory.yaml on each Mac, letting Exo establish a peer-to-peer mesh network via mDNS. This is the communication foundation of the entire cluster.

How mDNS and Exo Form the Network

mDNS (Multicast DNS) is the protocol behind Bonjour. It allows LAN devices to announce and discover each other's services via multicast addresses 224.0.0.251 (IPv4) or ff02::fb (IPv6) without relying on a central DNS server. After running exo cluster init, each Mac broadcasts its hostname, IP, and port under a service type like _exo._tcp.local. Other nodes listening on the same multicast domain can automatically establish peer-to-peer connections without manually entering every IP.

Exo relies on this mechanism to form a "mesh network": nodes can connect directly to one another, and model shard information and request state propagate through this network. This works out of the box in the same Layer 2 LAN such as a lab or home, but multicast is often blocked across subnets or on Wi-Fi with AP isolation enabled, so you'll need an explicit address list or ensure the network supports multicast. After the first network formation, it's recommended to verify service discovery with dns-sd -B _exo._tcp, and make sure all Macs' firewalls allow inbound traffic on the relevant ports—otherwise you'll see nodes but fail to establish a complete mesh.

Unified Memory: A Natural Shared Tensor Buffer

Apple's unified memory architecture lets the CPU and GPU address the same physical pages, completely eliminating PCIe copy overhead. This is the unique advantage of Apple Silicon for inference.

llama.cpp 0.3.0 leverages this feature via the --n-gpu-layers parameter. On an M2 Max, a 7B model can offload 35 layers (a 32-layer model can actually be over-allocated); and the M3 Ultra's 80-core GPU can host all 32 layers directly:

./llama-server -m model.gguf -c 4096 --n-gpu-layers 35 --port 8080

To verify layer residency, use: llama-server -m model.gguf --verbose 2>&1 | grep "gpu_layers".

Practical advice: set --n-gpu-layers to the maximum number of layers each machine's GPU memory can hold, pushing each piece of hardware to its limit.

Understanding --n-gpu-layers and "Layer Residency"

In inference engines like llama.cpp, LLMs execute sequentially by Transformer layer, with each layer containing attention, feed-forward network, and normalization modules. --n-gpu-layers N means placing the N layers of model parameters closest to the input on the GPU (Metal) for computation, with the remaining layers falling back to the CPU. Because unified memory lets the CPU and GPU share the same physical memory, these "GPU-offloaded" layers simply hand the computation to the GPU cores—no data copies are needed.

On Apple Silicon, how many layers you can offload depends on the Metal backend's recommended maximum working set size (recommendedMaxWorkingSetSize), which is typically a portion of total unified memory. If N is set close to or even slightly above the actual number of layers, the engine will place all layers on the GPU; the article's mention of M2 Max "35 layers, 32-layer models can be over-allocated" is because some non-layer weights also occupy VRAM, and the actual number of layers that fit is dynamically checked at runtime based on available memory. To verify, check the gpu_layers or offloaded N/N layers in the log to determine whether all are resident on the GPU. If overflow or fallback occurs, speed drops significantly, so "set it to the maximum number of layers GPU memory can hold" is the balance point between performance and stability.

Orchestration: Ray Handles Routing, Exo Handles Sharding

Ray handles request routing, while Exo is responsible for sharding the model across the entire mesh network.

Start the head on the most powerful node:

ray start --head --port=6379 --dashboard-host=0.0.0.0

Run on each worker node:

ray start --address=<head-ip>:6379 --resources='{"GPU": 1}'

Deploy a Ray Serve entry point that forwards each prompt to the actor with the shortest queue:

@serve.deployment(ray_actor_options={"num_gpus": 0.25})
class Llama:
    def __init__(self):
        self.llm = LlamaCPP(model_path="model.gguf", n_gpu_layers=35)
    async def __call__(self, request):
        return self.llm(request.json()["prompt"])

Note that num_gpus is set to 0.25, which allows a single GPU to host multiple actor instances simultaneously, improving resource utilization.

Ray Serve and the Meaning of num_gpus=0.25

Ray Serve is a model serving framework built on top of Ray. It turns each Python deployment into an HTTP-callable service and automatically handles request queuing, replica orchestration, and horizontal scaling. In @serve.deployment(ray_actor_options={"num_gpus": 0.25}), the num_gpus value is not a request for precise physical GPU partitioning; it's a resource quota declared to Ray's scheduler: an actor occupies only 0.25 "GPU units." This allows up to 4 actor replicas to reside on the same GPU simultaneously, improving utilization for small models with modest memory footprints or multi-instance inference.

However, num_gpus is only a logical declaration; it doesn't automatically isolate VRAM. What actually determines whether they can coexist is unified memory capacity and llama.cpp's memory usage. For a 7B model, if each LlamaCPP instance already occupies a large chunk of unified memory, launching multiple actors may instead cause them to compete and trigger OOM. Therefore, in practice, people often set num_gpus small and tune with replica count, batch size, and VRAM monitoring: start with a single replica running stably, then gradually increase replicas while watching GPU utilization and memory levels, avoiding a disconnect between resource declarations and actual usage.

Monitoring, Auto-Scaling, and Cost Accounting

Use Apple's native powermetrics to collect GPU utilization:

sudo powermetrics --samplers gpu_power -i 1000 -n 1

Feed the output into a custom Prometheus exporter (mac_gpu_exporter), scraping every 10 seconds.

Cost-wise, this is remarkable: four Macs draw about 120W total power on average. At the U.S. average electricity rate of about $0.015/kWh, running 24 hours costs less than $0.04. That's essentially free compute.

Add an alert rule to trigger auto-scaling:

avg_over_time(mac_gpu_utilization[5m]) > 0.8

Use a simple launchd job to wake sleeping Macs, achieving on-demand scaling.

Connecting powermetrics, Prometheus Exporters, and launchd

powermetrics is a low-level performance sampling tool bundled with macOS. It requires sudo privileges. --samplers gpu_power outputs GPU power, utilization, frequency, and other metrics; -i 1000 -n 1 means sample every 1000 milliseconds and exit after one sample. Unlike nvidia-smi on NVIDIA, macOS has no native, always-on HTTP endpoint for GPU metrics, so the article uses a custom component like mac_gpu_exporter to periodically run powermetrics, parse the text output, and convert it into a Prometheus-scrapeable /metrics format.

The other half of auto-scaling is launchd: it's macOS's init and service management daemon, equivalent to Linux's systemd. By writing plist files and placing them under ~/Library/LaunchAgents or /Library/LaunchDaemons, you can define jobs like "start Exo and Ray worker when invoked." Once the Prometheus alert rule fires, a lightweight controller calls launchctl load or SSH to wake the target Mac; when utilization drops, use launchctl unload to put it back to sleep. This way you don't need to keep all machines running, saving power while handling burst loads—but note that wake-up latency (seconds to tens of seconds) needs to match your service timeouts.

You Can Get It Running in One Evening

The barrier to entry for the whole process is lower than you'd think: run exo cluster init on each idle Mac, then point a single Ray Serve endpoint at the cluster, and by the next morning you have a usable distributed inference pool.

Several technical choices are worth noting: Exo handles model sharding, Ray handles scheduling, unified memory eliminates data transfer overhead, and powermetrics provides native monitoring—this combination makes full use of the unique capabilities of the Apple ecosystem rather than mechanically copying NVIDIA cluster approaches.

For individual developers and small teams, this is a pragmatic path to maximize the value of existing hardware assets.

One-line Summary

A complete hands-on path to turning idle Apple Silicon Macs into a distributed LLM inference cluster.

Paragraph Summary

This article walks through converting multiple idle Apple Silicon Macs in a lab into a distributed LLM inference cluster: first enumerate M2 Max, M3 Ultra, M1 Pro, and other devices on the LAN via arp-scan and record CPU/GPU/memory information, then form the network using Exo's mDNS peer-to-peer approach; leverage Apple unified memory to eliminate PCIe copies and offload as many model layers to the GPU as possible via llama.cpp's --n-gpu-layers; use Ray for request routing and Exo for model sharding, with Ray Serve scheduling by queue length and allowing a single GPU to host multiple actors; finally, monitor GPU utilization with powermetrics and Prometheus, implement on-demand auto-scaling based on alerts and launchd, and calculate that four Macs running all day cost less than $0.04 in electricity.