MoE
Mixture of Experts (MoE) is a machine learning architectural paradigm consisting of multiple specialized sub-networks called "experts" and a gating and routing mechanism. When processing each input, the routing mechanism activates only a subset of experts rather than all parameters, significantly reducing per-inference computation while maintaining the model's overall capacity. MoE is widely applied in large-scale language models and deep learning systems as a key technique for balancing model scale and computational efficiency.
Core Facts
Flash models typically use knowledge distillation technology, training smaller-parameter, faster-inference student models with flagship models as the teacher model.
The theoretical roots of the Mixture of Experts model can be traced back to the mixture of experts system proposed by Jacobs et al. in 1991.
DeepSeek-V3 adopts a Mixture of Experts (MoE) architecture with a total parameter count of 671 billion, but only approximately 37 billion parameters are activated per inference.
Timeline (last 90 days)
In the MoE architecture, after each token is input, the router selects the few highest-scoring experts to participate in computation, and the remaining experts are not activated.
Step 5 Preview's sparse activation design brings per-token compute close to the level of a 27B dense model while maintaining a large-capacity knowledge base.
The main bottleneck in MoE model serving lies in the irregular memory access and load imbalance caused by expert routing, which requires the scheduler and kernels to work together.
The Nemotron-H decoder uses 512-expert MoE layers, and at inference time dynamic routing activates only 22 experts, so of the 120B total parameters only 12B participate in computation per step.
Strata's core idea is to adopt a MoE (Mixture of Experts) architecture, where the model internally contains more than 20,000 small expert modules, and only a few relevant experts are invoked per inference.
With the maturation of the MoE architecture and advances in quantization technology, the inference threshold for hundred-billion-parameter models is moving from data centers down to personal devices.
MoE models with fewer activated parameters are prone to "breaking the chain" or skipping steps in long-chain reasoning, because the capacity of the activated experts is limited.
Understanding backpropagation is a prerequisite for understanding modern model architectures such as Transformer, GPT, LLaMA, and MoE.
The Strata engine uses an adaptive expert layer architecture (RemoteExpertOpt::adapt, PeerExperts::adapt), analogous to the MoE mixture-of-experts idea; when the required expert is not in VRAM, slots must be swapped between decoding windows from host memory or the peer, measured at about 1.1 milliseconds of pure waiting.
For production systems serving MoE models, the single decoding pipeline should be replaced with a prefill-decode disaggregated architecture, feeding the full-model KV cache directly into the pruned decoder.
40 more timeline events
All Facts (20)
Flash models typically use knowledge distillation technology, training smaller-parameter, faster-inference student models with flagship models as the teacher model.
90%VerifiedThe theoretical roots of the Mixture of Experts model can be traced back to the mixture of experts system proposed by Jacobs et al. in 1991.
90%VerifiedDeepSeek-V3 adopts a Mixture of Experts (MoE) architecture with a total parameter count of 671 billion, but only approximately 37 billion parameters are activated per inference.
90%VerifiedGoogle's Switch Transformer and Mistral's Mixtral 8x7B are typical representatives of the MoE architecture.
90%VerifiedThe MoE architecture typically activates only 10%–20% of total parameters during each inference
90%VerifiedThe MoE architecture divides a model's internals into multiple expert sub-networks, with a gating network dynamically selecting a small number of experts to participate in computation during each inference, so that the actual number of activated parameters is far smaller than the total parameter count.
90%VerifiedDeepSeek is a series of large language models developed by the Chinese artificial intelligence company Deepseek.
90%VerifiedThe VRAM usage and inference latency of MoE architectures are calculated based on the number of activated parameters, not the total parameter count.
90%VerifiedThe large-capacity, small-activation design of MoE allows a model's knowledge capacity and inference cost to be decoupled.
85%VerifiedMixture of Experts (MoE) models dynamically select a small number of experts to activate for each input token through a gating mechanism, thereby reducing the computational cost per inference while maintaining model capability.
85%VerifiedThe Mixture-of-Experts (MoE) architecture replaces feed-forward network layers with multiple expert sub-networks and dynamically activates a subset of experts via a router, so that the number of actually activated parameters is only a small fraction of the total.
85%VerifiedGoogle's Gemini 1.5 series has been confirmed to use a MoE (Mixture of Experts) architecture.
85%VerifiedGPT-4和Mixtral等主流大模型也采用了MoE架构
85%VerifiedQwen 3.8 Flash Next adopts a Mixture of Experts (MoE) architecture.
80%VerifiedDeepSeek V4 Flash is an MoE multimodal model with a total parameter count of 284B.
80%VerifiedDeepSeek V4 Flash is a cost-effective inference model released by DeepSeek, optimized for speed and cost.
80%VerifiedThe MoE architecture uses multiple expert networks and a routing mechanism to selectively activate only a subset of experts during each forward pass, reducing computational overhead while maintaining large model capacity.
80%VerifiedAll expert parameters of an MoE model need to reside in memory (even when not activated), which is also why ternary quantization is particularly important for MoE.
80%VerifiedThe core idea of MoE (Mixture of Experts) can be traced back to the early mixture of experts framework proposed by Jacobs et al. in 1991, and Google's Switch Transformer in 2021 brought it into the era of large-scale language models.
80%VerifiedPangu 2.0 Flash adopts a MoE (Mixture of Experts) architecture with a parameter scale of 92B A6B (92B total parameters, 6B activated parameters).
80%