[KongchangAI]
· 2 min read· 1,251 words

AI Supercomputer in a Backpack: DeepSeek Offline Inference Hits 220 tok/s in Real-World Test

AI Supercomputer in a Backpack: DeepSeek Offline Inference Hits 220 tok/s in Real-World Test

Two Mac Studios plus an e-bike battery run local LLM inference offline in a backpack at 220 tokens/s.

At SGLM, EXO Labs built a backpack cluster from two Mac Studios with 256GB unified memory each, powered by an e-bike battery, running DeepSeek V4.1 Flash offline at ~220 tokens/s. 512GB unified memory is required to fit the model; speed relies on Thunderbolt 5 RDMA cutting inter-machine latency to microseconds for practical tensor parallelism, plus MoE activating only some experts per token. The open-source EXO framework auto-handles device discovery, topology awareness, and model sharding.

At the SGLM conference, the EXO Labs team couldn't secure a booth—so they stuffed two Mac Studios into a backpack and created a "walking booth." This backpack setup runs DeepSeek V4.1 Flash locally and offline at roughly 220 tokens per second. That number isn't remarkable in the cloud, but on a device powered by an e-bike battery and carried on your back, it means something entirely different.

256GB unified memory per machine

Hardware Configuration: Two Mac Studios, Brute-Force Style

The core of this setup is two M5 Ultra Max Studios, each with 256GB of unified memory—512GB total. Power comes from an e-bike battery, so the whole rig can be carried anywhere and run untethered from a wall outlet.

512GB of unified memory is the prerequisite for getting this to run at all. DeepSeekAd V4.1 Flash's model size requires a sufficiently large memory pool to fit entirely; a single 256GB machine isn't enough, and two spliced together just barely cross the threshold. This also explains why the team chose a "dual-machine" approach rather than a single machine—it's not performance redundancy, it's a hard requirement for memory capacity.

Inference speed of roughly 220 tokens per second

How 220 Tokens Per Second Is Achieved

Reaching roughly 220 tokens per second isn't just about having lots of memory. The key factors EXO Labs highlighted in their demo span three levels.

Thunderbolt 5 RDMA: Pushing Inter-Machine Latency Down to a Few Microseconds

For two machines to collaborate on inference, inter-machine communication latency is the biggest bottleneck. This setup uses Thunderbolt 5 RDMA to bring inter-machine latency down to the microsecond level. Latency at this scale turns tensor parallelism from "theoretically feasible" into "practically usable"—if cross-machine communication were an order of magnitude slower, the gains from tensor parallelism would be completely eaten up by communication overhead.

Thunderbolt 5 RDMA brings inter-machine latency down to a few microseconds

RDMA (Remote Direct Memory Access) is a technology that lets one machine directly read and write another machine's memory. Data transfer bypasses both machines' operating systems and CPU protocol stacks, handled directly by the network card or interface controller. Traditional network communication goes through multiple layers of copying from user space to kernel space plus interrupt handling, typically resulting in tens to hundreds of microseconds of latency; RDMA compresses this path, bringing latency down to single-digit microseconds. In multi-machine tensor-parallel inference, every token generated requires synchronizing intermediate activations or gradient shards across machines—communication happens extremely frequently, so inter-machine latency almost directly determines overall throughput. Thunderbolt 5 provides high-bandwidth, low-latency point-to-point connections, and with RDMA semantics, data exchange between the two Mac Studios approaches the experience of shared memory. This is one of the underlying conditions that allows the dual-machine setup to reach 220 tokens per second.

MoE Architecture: Each Token Activates Only Some Experts

DeepSeek V4.1 Flash uses a MoE (Mixture of Experts) architecture, where each token activates only some experts rather than mobilizing all parameters. This dramatically reduces the computation per inference pass and is another pillar of the speed performance. Large memory handles "fitting it in," MoE handles "computing efficiently," and only the combination of both produces 220 tokens per second.

MoE (Mixture of Experts) splits the model's feed-forward layers into many parallel "expert" sub-networks, paired with a routing network that selects only a few experts per token to participate in computation. This allows the model's total parameter count to be very large for greater capacity, while the actual computation per forward pass depends only on the number of activated experts, not the total parameter count. The trade-off is that memory usage is still calculated based on total parameters—all experts must fit in memory, which is exactly why 512GB of unified memory is a hard requirement. In other words, MoE separates "computing efficiently" from "storing large": the compute side uses sparse activation to reduce FLOPs, while the storage side still needs to fully accommodate all expert weights. Understanding this is essential to grasping why this setup both needs large memory and can achieve high throughput.

Tensor parallelism is a technique that splits a single Transformer layer's weight matrices across multiple devices by dimension, with each device holding and computing only a portion, then aggregating partial results through collective communication (such as All-Reduce). It's the standard distributed strategy when a model is too large to fit on a single card. Unlike pipeline parallelism, which splits by layer, tensor parallelism generates communication within every layer, making it extremely sensitive to inter-device bandwidth and latency. Once communication overhead exceeds the computational gains from parallelism, the speedup ratio drops sharply. This is precisely why the earlier emphasis on microsecond-level latency matters: only when cross-machine communication is fast enough can tensor parallelism go from "theoretically feasible" to "practically usable," and only then does splicing two machines truly equate to one larger machine.

EXO: Turning Multiple Apple Silicon Devices into One Machine

The open-source project behind this demo is EXO, a distributed inference framework. Its value lies in automation: device discovery, topology awareness, and model sharding are all handled by the framework—users don't need to manually configure which machine runs which layer.

Can automatically assemble multiple Apple Silicon devices into a local cluster

EXO can automatically assemble multiple Apple Silicon devices into a local cluster. This means the approach isn't limited to extreme configurations like "two Mac Studios"—in theory, any multiple Mac devices on hand can be aggregated into an inference cluster. For developers and teams who need to run large models locally without relying on the cloud, this offers a replicable path.

What This Setup Means

The most direct significance of compressing data-center-level inference capability into a backpack is privacy and offline operation. Neither models nor data leave the local environment, with no network dependency—suitable for data-sensitive scenarios.

Another notable point is the cost structure. Using consumer-grade hardware plus an open-source framework to assemble a usable inference cluster bypasses the cloud's per-token billing model. Of course, the hardware investment for this configuration itself isn't low—two high-spec Mac Studios plus a battery is more of a technical demonstration than a mainstream solution. But the direction it validates is clear: the engineering feasibility of local distributed inference has been proven, and the remaining challenges are scale and ease of use.

For ordinary users, the more realistic short-term approach might be to use EXO to cluster the few Apple Silicon devices they already own, rather than replicating this backpack setup. The framework itself is open source; the barriers are mainly hardware and tuning.

One-line Summary

Two Mac Studios plus an e-bike battery, running local LLM inference offline in a backpack at 220 tokens per second.

Paragraph Summary

At the SGLM conference, having failed to secure a booth, EXO Labs assembled a backpack-style local cluster from two Mac Studios each with 256GB of unified memory, powered by an e-bike battery, running DeepSeek V4.1 Flash offline at roughly 220 tokens per second. 512GB of unified memory is the hard prerequisite for fully loading the model—splicing two machines is a capacity necessity, not performance redundancy. Speed depends on Thunderbolt 5 RDMA reducing inter-machine latency to a few microseconds, making tensor parallelism practically usable, plus the MoE architecture activating only some experts per token to reduce computation. The underlying open-source framework EXO automatically handles device discovery, topology awareness, and model sharding, aggregating multiple Apple Silicon devices into a local cluster. The setup validates the engineering feasibility of local distributed inference, with value in privacy, offline operation, and bypassing the cloud's per-token billing.

#EXO Labs#DeepSeek V4.1 Flash#local LLM inference#Mac Studio cluster#Thunderbolt 5 RDMA#MoE architecture#distributed inference
Share:

Related articles