Lookahead Expert Scheduling
An expert scheduling technique for on-device inference of MoE models that predicts the experts required by the next layer and preloads their weights, hiding flash-memory read latency behind computation time.
Timeline (last 90 days)
The LES scheme reduced expert weight size by 26.7%, while the average score across five benchmarks dropped by only 0.16 points.
Lookahead Expert Scheduling (LES) technology, without changing the model's actual routing logic, hides the bottleneck of moving expert weights from flash storage to computation by predicting the experts likely needed by the next layer and preloading their weights.
On the iPhone 18 Pro Max, the LES scheme achieved a throughput of 20.51 tok/s, a 6.07x speedup over the 3.38 tok/s baseline.
In the past, optimization of on-device large models focused mostly on quantization and pruning, but system-level scheduling and prefetching can likewise unlock enormous headroom.
The core mechanism of LES (predicting the next layer's expert needs and preloading) is not tied to a specific model architecture; as long as an MoE model's routing pattern is somewhat predictable, similar scheduling strategies can be transferred to other MoE models.
DSpark is a speculative decoding implementation that works in concert with LES's expert preloading to further amplify on-device throughput gains.
MiniCPM-V 4.7 35B-A3B uses a combination of three techniques—Lookahead Expert Scheduling (LES), fine-grained layer-by-layer quantization, and DSpark speculative sampling—to accelerate on-device inference.
All Facts (7)
The LES scheme reduced expert weight size by 26.7%, while the average score across five benchmarks dropped by only 0.16 points.
50%UnverifiedLookahead Expert Scheduling (LES) technology, without changing the model's actual routing logic, hides the bottleneck of moving expert weights from flash storage to computation by predicting the experts likely needed by the next layer and preloading their weights.
50%UnverifiedOn the iPhone 18 Pro Max, the LES scheme achieved a throughput of 20.51 tok/s, a 6.07x speedup over the 3.38 tok/s baseline.
50%UnverifiedIn the past, optimization of on-device large models focused mostly on quantization and pruning, but system-level scheduling and prefetching can likewise unlock enormous headroom.
50%UnverifiedThe core mechanism of LES (predicting the next layer's expert needs and preloading) is not tied to a specific model architecture; as long as an MoE model's routing pattern is somewhat predictable, similar scheduling strategies can be transferred to other MoE models.
50%UnverifiedDSpark is a speculative decoding implementation that works in concert with LES's expert preloading to further amplify on-device throughput gains.
50%UnverifiedMiniCPM-V 4.7 35B-A3B uses a combination of three techniques—Lookahead Expert Scheduling (LES), fine-grained layer-by-layer quantization, and DSpark speculative sampling—to accelerate on-device inference.
50%