No More Prefetching: Bevy Uses Request Interleaving to Speed Up MoE Inference by 3.46x

When running large Mixture of Experts (MoE) language models locally, expert offloading is a key technique that allows infrequently used expert parameters to be stored in external storage (such as CPU memory or SSD) and loaded onto the GPU only when needed. However, traditional batching causes the GPU to stall while waiting for missing experts, severely hurting inference speed.

The Limits of Prefetching: The Bigger the Cache, the Harder the Prediction

To hide expert loading latency, a common remedy is prefetching: predicting which experts will be needed next and using idle link bandwidth to load them in advance. But the effectiveness of this approach depends heavily on prediction accuracy. Imprecise predictions waste bandwidth and GPU memory fetching experts that are not actually needed.

The key finding of this research is: when an expert cache exists in GPU memory, prefetching can be harmful. This is because as the cache contains more of the required experts, the accuracy of expert prediction actually declines. In other words, the larger the cache, the smaller the benefit of prefetching—and it may even drag down performance by occupying precious GPU memory.

The core structure of a Mixture of Experts (MoE) model replaces the traditional dense feed-forward network with multiple parallel "expert" subnetworks and introduces a gating network that activates only a few experts per token (typically Top-2). As a result, the full parameters of one MoE layer can reach tens or even hundreds of GB, while only a small fraction is actually used in a single forward pass. Expert offloading exploits exactly this sparse activation property: leaving inactive experts in CPU memory or on SSD and moving only the experts needed by the current batch to the GPU. This mechanism lets consumer-grade devices with limited VRAM run very large models, but at the cost of moving experts over bandwidth-constrained links such as PCIe. If the transfer cannot overlap with computation, the GPU idles while waiting, creating what is known as an "expert loading stall." Understanding this is the prerequisite for understanding why both prefetching and scheduling revolve around "hiding loading latency."

Core Insight: Proper Request Scheduling Beats Prefetching

Based on the above finding, the researchers put forward a key insight: proper request scheduling is more effective than prefetching. Rather than working hard to predict future expert demand, it is better to naturally reveal expert demand by scheduling concurrent requests and overlapping computation with loading.

Prefetching is essentially an optimization in the "time dimension": before an expert is actually needed, it speculates about future demand based on historical access patterns or lightweight predictors and initiates the transfer in advance. Its benefit depends entirely on the prediction hit rate—if the prediction is right, loading latency is fully hidden; if wrong, link bandwidth and limited GPU memory are wasted, and experts that are actually needed may even be evicted from the cache. Request scheduling, by contrast, is an optimization in the "spatial dimension": by serving multiple concurrent requests at once, their expert demands naturally stagger in time, so when one request is blocked by a missing expert, another request whose experts are already ready can continue computing. This way, without any prediction, expert demand is "exposed" by the requests themselves. Bevy's contribution is to engineer this intuition: through runahead inference, ready requests are allowed to move past blocked layers, and a self-balancing scheduler controls the overhead of repeatedly reading shared weights, thereby achieving overlap of computation and loading without prediction.

Bevy: A Concurrent Request Scheduling System Based on Runahead Inference

Based on this insight, the researchers designed Bevy, an MoE inference system based on runahead inference and specifically optimized for concurrent requests.

How It Works: Letting Ready Requests "Run Ahead"

Bevy's core mechanism is: when the experts required by a request are already available, that request can "run ahead" to later layers while other requests wait for experts to load. In this way, Bevy can reveal which experts the request will need next, fetching those missing experts while computing the ready request—achieving overlap of computation and loading without predicting expert selection.

Self-Balancing Scheduler: Balancing Benefit and Overhead

Running requests individually would repeatedly read shared weights from GPU memory, increasing computation time. To limit this overhead, Bevy designs a self-balancing scheduler. At each layer, it batches ready requests and prioritizes lagging requests, dynamically adjusting the grouping according to changes in expert availability.

llama.cpp is one of the most popular local LLM inference frameworks today, known for its pure C/C++ implementation, support for hybrid CPU/GPU inference, and multiple quantization formats, and is often used as the default baseline and performance reference for local deployment. In expert offloading scenarios, llama.cpp executes layers sequentially and blocks whenever an expert is missing, so GPU utilization fluctuates noticeably with the expert hit rate. Bevy chose llama.cpp as the end-to-end comparison baseline, measuring whole-machine decode throughput (tokens/s), a metric that reflects both computational efficiency and link utilization. The "fastest prefetching baseline" represents the best solution using predictive prefetching on the same hardware; Bevy achieves a 1.55x improvement over it, showing that its advantage comes not from faster transfers but from fewer useless transfers and better computation/loading overlap.

Performance: Significantly Outperforming Baselines

On two local platforms, Bevy's decode throughput reached 3.46x that of llama.cpp and 1.55x that of the fastest prefetching baseline. This result validates the idea that request scheduling outperforms prefetching and offers a new direction for accelerating local MoE inference.

Bevy's success shows that in expert offloading scenarios, rather than relying on imperfect predictions, it is better to exploit concurrency across requests through clever scheduling, thereby using GPU and link resources more efficiently. This approach is expected to further promote the adoption of local large-model inference.