MoEShift
A dynamic load scheduling strategy for Mixture-of-Experts (MoE) models that reduces repeated loading and communication costs through expert migration, caching, and load balancing, thereby improving inference throughput.
Timeline (last 90 days)
A Hacker News discussion reports that an M5 Pro (24GB) running Qwen 3.8 Flash Next reaches a prefill speed of 374 Tok/S.
MoEShift is a dynamic load scheduling strategy for Mixture-of-Experts (MoE) models, aimed at distributing compute tasks more efficiently across available hardware resources.
24GB of memory falls in the "sufficient but not ample" range for running medium-scale MoE models; if MoEShift can dynamically manage expert residency, it may sustain relatively high throughput under limited memory.
The prefill performance improvement of the M5 Pro over the M4 Pro may come partly from higher chip compute and partly from software-layer optimizations such as MoEShift.
All Facts (4)
A Hacker News discussion reports that an M5 Pro (24GB) running Qwen 3.8 Flash Next reaches a prefill speed of 374 Tok/S.
50%UnverifiedMoEShift is a dynamic load scheduling strategy for Mixture-of-Experts (MoE) models, aimed at distributing compute tasks more efficiently across available hardware resources.
50%Unverified24GB of memory falls in the "sufficient but not ample" range for running medium-scale MoE models; if MoEShift can dynamically manage expert residency, it may sustain relatively high throughput under limited memory.
50%UnverifiedThe prefill performance improvement of the M5 Pro over the M4 Pro may come partly from higher chip compute and partly from software-layer optimizations such as MoEShift.
50%