Turning Your iPhone into a Second GPU for MacBook: 29–44% Faster Local LLM Prefill

A developer shared an imaginative local inference setup on Reddit: using a 10 Gb/s USB-C cable to connect an iPhone 17 Pro Max to a 24 GB M4 Pro MacBook, effectively turning the phone into a second GPU to share the computational load of a locally running Qwen 3.8 27B (IQ4_XS quantized) model. The result: prefill speed improved by 29%–44%, with the phone also taking on some of the storage burden for the context window.

This isn't a benchmark stunt — it's an engineering solution targeting a real pain point in local Agent workflows. On a 24 GB Mac, after loading the model, only enough memory remains for roughly 64k tokens of 8-bit context, meaning every file read or tool result introduces waiting. The author put the idle A19 Pro chip sitting in their pocket to work.

Core Idea: Layer Splitting with Pipeline Parallelism

The key to this approach is splitting the Transformer's layers and running them in a pipeline across two devices. The Mac handles layers 1–40 for each 256-token batch, then streams the resulting activations to the phone over USB-C; the phone runs layers 41–64 on its own GPU, while the Mac has already started processing the next batch.

What truly makes this viable is the matrix operation units built into the A19 Pro GPU (Metal 4 tensor ops). The author notes that with these tensor cores, the phone's half of the computation is 2.4× faster than without them — this is the hardware prerequisite that makes it worthwhile to pull the phone into the inference pipeline.

Activations are the intermediate tensors output by each layer of a neural network after computation. In a Transformer, each layer's output becomes the next layer's input, so the "data" transferred between layers is these activations. For a 256-token batch, the activation size is approximately batch_size × sequence_length × hidden_dim × bytes_per_dtype. For a 27B-scale model with a common hidden dimension of 5120 and FP16 precision, a single batch's activations are about 1.3 MB — well within the acceptable range for USB-C 3.2 Gen 2 (theoretical 10 Gb/s, effective bandwidth ~800 MB/s), with a latency of only ~1.6 ms, which won't bottleneck the pipeline. The core advantage of pipeline parallelism is that while the phone processes layers 41–64, the Mac can simultaneously begin processing the next batch's layers 1–40 — the two segments of computation overlap in time, reducing throughput loss from waiting.

Benchmark Results: Longer Context, Greater Gains

The author compared "phone off" vs. "phone on" within the same build, prefilling a 2,000-token file into an already-saved Agent session, and obtained the following end-to-end (E2E) prefill rates:

• 8k context: Mac alone 132 tok/s → Mac + iPhone 177 tok/s (+35%)
• 16k context: 109 tok/s → 157 tok/s (+44%)
• 32k context: 101 tok/s → 130 tok/s (+29%)
• 48k context: 87 tok/s → 113 tok/s (+30%)

In a cold-start scenario with a fresh 27k-token session: the original llama.cpp takes 245 seconds, the author's fork running on Mac alone takes 228 seconds, and with the phone connected it drops to 168 seconds.

It's worth noting an honest correction from the author: the prefill TPS displayed on the phone's screen only counts the layers it handles, not the end-to-end rate. He's working on fixing this display issue. The numbers listed above are accurate end-to-end prefill rates.

Beyond 64k: Phone Shifts from "Compute" to "Memory"

The cleverest part of the design is what happens when context exceeds 64k — the phone's role changes. Instead of running layers 41–64, it takes over the older KV cache: the earliest KV pages are migrated to the phone, and the Mac runs all 64 layers itself.

For each attention layer, the phone runs attention over the old keys on its own GPU, and the Mac merges those results with its own portion. Going further, during the writing phase, the phone's Neural Engine is also engaged — each 16k-key chunk of old context is compiled into a NE model with keys as weights. The author reports that at 140k context, this reduces per-token write time from 279 ms to 176 ms (compared to phone GPU only).

The server allocates 196k–229k of 8-bit context based on the phone's available memory, which is equivalent to offloading up to ~5.7 GB of KV cache to the phone, causing the Mac's memory usage to stop growing at 64k. The author tested a session that grew to 128k (8-bit) and successfully recalled all 3 pre-planted facts; at 140k (4-bit), greedy decoding output was completely identical to a pure Mac run across 32 generated tokens.

KV Cache (Key-Value Cache) is the primary source of memory overhead during Transformer inference. In autoregressive generation, each attention layer must store the Key and Value vectors for all historical tokens to avoid recomputation. Its size grows linearly with context length: context_length × num_layers × (K_dim + V_dim) × bytes_per_dtype. For a 27B model with 64 layers and 8-bit quantization, each 1k tokens requires roughly 0.5–1 MB, so 64k context already consumes ~32–64 GB — far beyond what a typical Mac can spare. This is exactly why a 24 GB MacBook only has room for ~64k of context after loading the model — the model weights themselves consume most of the memory. Offloading old KV pages to the phone is essentially trading USB-C bandwidth for context window expansion beyond the local memory limit. Neural Engine is a hardware module in Apple Silicon specifically designed for fixed-pattern operations like matrix multiplication and convolution. It has extremely high throughput but less flexibility than a GPU, making it ideal for batch-executing repetitive attention computations compiled as fixed-weight models.

Limitations: No Write Speed Gains Below 64k

The author is candid about the boundaries of this system. Below 64k, the phone doesn't help with generation speed — that's the Mac's job. What actually pushed generation speed from llama.cpp's baseline 11.3 tok/s to ~25 tok/s at ~30k context are custom kernels in the fork: SME2 instructions on the M4 CPU, Metal fused operators, and DFlash2 speculative decoding. SME2 alone can contribute up to an additional 29% prefill improvement on the Mac side.

In other words, the phone's contribution is concentrated in two scenarios: the prefill phase (the phone only joins for ~512+ tokens) and old-key attention computation for very long contexts. In one real session, only 7 out of 36 requests triggered phone involvement — but those 7 covered ~83% of the tokens read. The current implementation also handles only one request at a time, and beyond 64k, the phone can't simultaneously act as both "storage" and "compute" — the author says this is the next problem to solve.

SME2 (Scalable Matrix Extension 2) is an Arm architecture CPU extension instruction set introduced for high-performance matrix operations, first appearing in the processor cores used in Apple M4. It allows the CPU to efficiently perform matrix multiplication without relying on the GPU, significantly accelerating linear layers in LLM inference (especially the matrix-vector multiplications during single-token generation). DFlash2 is a speculative decoding implementation the author built into the fork: speculative decoding uses a small draft model to generate multiple candidate tokens in advance, then has the main model verify them in parallel, outputting multiple tokens per forward pass and breaking the serial bottleneck of autoregressive token-by-token generation. Both of these optimizations are unrelated to the phone — they are purely Mac-side kernel improvements, which explains why the generation speed gains below 64k (11.3 → 25 tok/s) should not be attributed to the iPhone.

Looking Ahead: New Architectures + New Hardware

The author notes that the real "gold mine" lies in newer model architectures paired with newer phone hardware. For example, DeepSeek^Ad V4.1-Flash's global KV cache requires only 890 bytes per token and introduces n-gram embedding tables (Engram); Qwen3.8-Flash-Next (a Qwen 4 architecture preview) also features n-gram lookup tables. None of these are present in the 27B model he tested, and he hasn't benchmarked either architecture yet.

On the hardware side, he's also anticipating that the A20 Pro in the iPhone 18 Pro Max could unlock even more headroom. The project's code, configurations, and benchmark scripts are open-sourced on GitHub (StayLameBro/backburner), and the author mentions the entire system was built with the help of Opus 5.5.

For local LLM enthusiasts, the value of this approach isn't in absolute speed — it's in showcasing a widely overlooked path: when memory becomes the hard constraint in local inference, layer splitting and KV offloading across heterogeneous devices may be more practical than simply throwing more single-machine compute at the problem.