MoE
Mixture of Experts (MoE) is a machine learning architectural paradigm consisting of multiple specialized sub-networks called "experts" and a gating and routing mechanism. When processing each input, the routing mechanism activates only a subset of experts rather than all parameters, significantly reducing per-inference computation while maintaining the model's overall capacity. MoE is widely applied in large-scale language models and deep learning systems as a key technique for balancing model scale and computational efficiency.
Core Facts
Flash models typically use knowledge distillation technology, training smaller-parameter, faster-inference student models with flagship models as the teacher model.
The theoretical roots of the Mixture of Experts model can be traced back to the mixture of experts system proposed by Jacobs et al. in 1991.
DeepSeek-V3 adopts a Mixture of Experts (MoE) architecture with a total parameter count of 671 billion, but only approximately 37 billion parameters are activated per inference.
Timeline (last 90 days)
On Mac unified memory, MoE models still need to load all weights into memory; having fewer active parameters only affects the amount of computation, not memory usage.
The AWS engineering team carried out kernel-level optimizations on Amazon SageMaker AI for a fused MoE model, including Swap-AB to improve WGMMA utilization, Triton backend routing to implement FP8 quantization, and eliminating repeated Triton JIT compilation.
The three kernel optimizations can be orthogonally stacked with upper-layer optimizations, and for teams deploying MoE models on SageMaker AI, these optimizations will likely be reflected in service performance in a managed form.
MoE's dynamic routing causes the number of tokens received by each expert and each batch to change constantly, making shape combinations highly prone to divergence, which leads to a low Triton compilation cache hit rate and thus causes long-tail latency and first-token jitter.
The AWS team used Triton's backend routing mechanism to let different computation paths go to the backend implementations suited to each, thereby maintaining numerical stability under FP8 quantization.
On the Hopper architecture, general-purpose kernels in MoE inference struggle to saturate tensor core compute, because WGMMA has strict requirements on matrix layout and pipelining, and a mismatch in memory access patterns or data layout causes a noticeable drop in utilization.
Unsloth v0.1.907-beta officially claims inference speed improvements of up to about 5x, mainly from a cache hit rate of about 95% and Strata-style fixes for MoE architectures.
Qwen3-Next-80B has a MoE architecture with about 80B total parameters, but each forward inference only activates about 3B parameters.
MoE models contain multiple independent expert feed-forward blocks, and each generated token is routed to only a few experts, for example 8 out of 256.
Under high power operation, the 35B MoE model reaches 78 to 107 tokens/s, while the dense 27B model only reaches 22 tokens/s.
40 more timeline events
All Facts (20)
Flash models typically use knowledge distillation technology, training smaller-parameter, faster-inference student models with flagship models as the teacher model.
90%VerifiedThe theoretical roots of the Mixture of Experts model can be traced back to the mixture of experts system proposed by Jacobs et al. in 1991.
90%VerifiedDeepSeek-V3 adopts a Mixture of Experts (MoE) architecture with a total parameter count of 671 billion, but only approximately 37 billion parameters are activated per inference.
90%VerifiedGoogle's Switch Transformer and Mistral's Mixtral 8x7B are typical representatives of the MoE architecture.
90%VerifiedThe MoE architecture typically activates only 10%–20% of total parameters during each inference
90%VerifiedThe MoE architecture divides a model's internals into multiple expert sub-networks, with a gating network dynamically selecting a small number of experts to participate in computation during each inference, so that the actual number of activated parameters is far smaller than the total parameter count.
90%VerifiedDeepSeek is a series of large language models developed by the Chinese artificial intelligence company Deepseek.
90%VerifiedThe VRAM usage and inference latency of MoE architectures are calculated based on the number of activated parameters, not the total parameter count.
90%VerifiedThe large-capacity, small-activation design of MoE allows a model's knowledge capacity and inference cost to be decoupled.
85%VerifiedMixture of Experts (MoE) models dynamically select a small number of experts to activate for each input token through a gating mechanism, thereby reducing the computational cost per inference while maintaining model capability.
85%VerifiedThe Mixture-of-Experts (MoE) architecture replaces feed-forward network layers with multiple expert sub-networks and dynamically activates a subset of experts via a router, so that the number of actually activated parameters is only a small fraction of the total.
85%VerifiedGoogle's Gemini 1.5 series has been confirmed to use a MoE (Mixture of Experts) architecture.
85%VerifiedGPT-4和Mixtral等主流大模型也采用了MoE架构
85%VerifiedQwen 3.8 Flash Next adopts a Mixture of Experts (MoE) architecture.
80%VerifiedDeepSeek V4 Flash is an MoE multimodal model with a total parameter count of 284B.
80%VerifiedDeepSeek V4 Flash is a cost-effective inference model released by DeepSeek, optimized for speed and cost.
80%VerifiedThe MoE architecture uses multiple expert networks and a routing mechanism to selectively activate only a subset of experts during each forward pass, reducing computational overhead while maintaining large model capacity.
80%VerifiedAll expert parameters of an MoE model need to reside in memory (even when not activated), which is also why ternary quantization is particularly important for MoE.
80%VerifiedThe core idea of MoE (Mixture of Experts) can be traced back to the early mixture of experts framework proposed by Jacobs et al. in 1991, and Google's Switch Transformer in 2021 brought it into the era of large-scale language models.
80%VerifiedPangu 2.0 Flash adopts a MoE (Mixture of Experts) architecture with a parameter scale of 92B A6B (92B total parameters, 6B activated parameters).
80%