iPhone Runs 35B MoE at 20.5 tok/s: The Scheduling Magic That Skips No Experts

Running large models locally on phones is nothing new, but it usually comes at the cost of model compression, expert pruning, or significant accuracy loss. OpenBMB's latest engineering practice offers a different approach: running a 35B-A3B MoE model on an iPhone at 20.5 tok/s without skipping a single expert.

According to Reddit community discussions and OpenBMB's engineering blog, MiniCPM-V 4.7 leverages a technique called Lookahead Expert Scheduling (LES) to hide the bottleneck of moving expert weights from flash storage to compute, all while keeping the model's actual routing logic unchanged.

Core Idea: Hiding Memory Transfers Behind Computation

The bottleneck in traditional on-device MoE inference often isn't compute power—it's storage bandwidth. Expert weights are enormous and can't all reside in memory, so every time routing selects a new expert, it must be loaded from flash. The common workaround is to shrink the model or skip some experts entirely, but this changes model behavior and degrades output quality.

LES takes a "predictive" approach: it anticipates which experts the next layer will need, preloads their weights from flash into memory while the current layer is computing, and prioritizes executing experts that are already ready. This way, flash read latency is masked by computation time, and the set of experts the model actually executes is identical to the original routing.

Combined with optimized weight layout, mixed-precision quantization, and DSpark speculative decoding, this solution achieves 20.51 tok/s throughput on an iPhone 18 Pro Max—a 6.07x speedup over the 3.38 tok/s baseline.

MoE (Mixture of Experts) architecture is the prerequisite for understanding this article. Unlike dense models that activate all parameters for every token, MoE models consist of multiple "expert" subnetworks plus a routing network, where each token is routed to only a few experts for computation. The "35B-A3B" mentioned here means approximately 35B total parameters with only about 3B activated per token. This gives the model large total capacity with small per-inference compute, making it ideal for resource-constrained edge devices. But the trade-off is that the total expert weight volume remains enormous and can't all reside in phone memory, while routing changes dynamically—causing frequent flash retrieval of different experts during inference and creating a storage bandwidth bottleneck. This is precisely the core conflict LES aims to solve.

Storage bandwidth bottleneck (memory wall) refers to the situation where compute units process data far faster than data can be moved from memory/flash, making actual system performance limited by data transfer rather than compute. This is especially pronounced on edge devices: flash random read latency is much higher than memory, and each MoE routing to a new expert may trigger a flash load. If "load—compute" is executed serially, the GPU/CPU sits idle for extended periods. Prefetching and scheduling are classic techniques to alleviate this bottleneck: moving subsequent data ahead while waiting for current computation, overlapping transfer with compute, and thereby "hiding" the bandwidth bottleneck behind computation time.

Performance Data: 6x Faster, Only 0.16 Points Accuracy Loss

From the published data, LES delivers substantial gains:

• Throughput improvement: 20.51 tok/s on iPhone 18 Pro Max, 6.07x the 3.38 tok/s baseline.
• Size reduction: Expert weights shrunk by 26.7%, with only a 0.16-point average drop across five benchmarks.
• Memory usage: Peak memory around 3 GB in low-memory configuration.

The significance of these numbers is that they prove running large MoE models on-device doesn't have to sacrifice routing integrity. A 0.16-point accuracy loss is nearly imperceptible in most application scenarios, while the 6x throughput improvement directly determines whether the interactive experience is usable.

Speculative decoding is a technique that accelerates autoregressive generation without changing the output distribution. It first uses a cheap small model (draft model) to rapidly generate several candidate tokens, then has the large model verify these candidates in parallel in one pass; accepted tokens are kept directly, while rejected ones are rolled back and recomputed. Since the large model can verify multiple tokens in a single forward pass, this effectively reduces the number of expensive model forward passes, improving throughput. DSpark is the speculative decoding implementation integrated into this solution, and it works in concert with LES's expert preloading to further amplify on-device throughput gains.

Mixed-precision quantization refers to using different numerical precisions for different parts of the model (e.g., 4-bit weights, 8-bit or 16-bit activations) to balance size compression and accuracy preservation. The "26.7% expert weight reduction with only 0.16-point average benchmark drop" is the combined result of quantization and weight layout optimization: expert weights constitute the bulk of model size, so quantizing them significantly reduces flash footprint and transfer volume, while accuracy loss is kept within acceptable bounds.

Open-Source Availability and Portability

According to OpenBMB, the implementation—including expert scheduling and DSpark integration—has been open-sourced in llama.cpp-omni. This means other developers can directly reuse this scheduling logic.

More noteworthy is the method's generality. The core of LES is "predicting next-layer expert demand and preloading," a mechanism not tied to any specific model architecture. In theory, as long as a MoE model's routing patterns have some degree of predictability, similar scheduling strategies can be ported to other MoE models, providing an optimization path for on-device deployment that doesn't sacrifice model integrity.

llama.cpp is an efficient large model inference framework implemented in C/C++, targeting consumer hardware and edge devices. It supports multiple quantization formats and cross-platform deployment, making it one of the mainstream choices for running models locally on phones and laptops. llama.cpp-omni can be understood as an engineering branch extended on top of it, integrating multimodal and expert scheduling capabilities. Integrating LES and DSpark into such an open-source framework means the scheduling logic is published as reusable engineering code—other developers can experiment with similar optimizations in their own MoE deployments without implementing prediction and prefetch mechanisms from scratch. This is the practical foundation for the solution's portability.

Implications for On-Device AI

In the past, on-device large model optimization focused mainly on quantization and pruning, but LES reminds us that system-level scheduling and prefetching can also unlock enormous headroom. When model weights can't all fit in memory, how to efficiently move data between storage and compute may be more valuable than simply compressing the model.

For teams looking to deploy MoE models on phones, tablets, and other devices, this "skip no experts" scheduling approach offers a new direction worth considering.

(This article is based on Reddit community discussions and OpenBMB's public engineering materials.)