Unverified50% confidenceFactExact time
The bottleneck in traditional on-device MoE inference often lies in storage bandwidth rather than compute, because the enormous size of expert weights prevents them all from residing in memory, and each routing to a new expert requires loading from flash storage.
1
Sources
50%
Confidence
Long-term
Relevance
10/21/2026
First Seen
Sources
iPhone Runs 35B MoE at 20.5 tok/s: The Scheduling Magic That Skips No Experts
redditr/LocalLLaMA10/21/2026
Related Entities
Related Claims
UnverifiedThe main bottleneck in MoE model serving lies in the irregular memory access and load imbalance caused by expert routing, which requires the scheduler and kernels to work together.76% similarUnverifiedThe MoE architecture faces the problem of uneven expert load, with vastly different activation frequencies among different experts, easily causing some devices to be idle while others are overloaded.75% similarUnverifiedThe Top-K routing strategy in MoE is prone to causing a small number of experts to be overused (the expert collapse problem), which requires techniques such as auxiliary load-balancing loss functions to address.75% similarVerifiedAlthough MoE models infer quickly, they have higher demands on VRAM and memory capacity because all expert parameters must be loaded, representing a trade-off between speed and capacity.73% similarUnverifiedThrough sparse activation, MoE uses a routing network to select the Top-K experts (typically K=2) for computation when processing each token, while the remaining experts do not participate in the computation.73% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/1081664API
curl https://kongchang.com/api/v1/knowledge/claims/1081664MCP
get_claim(id=1081664)