MoE
Mixture of Experts (MoE) is a machine learning architectural paradigm consisting of multiple specialized sub-networks called "experts" and a gating and routing mechanism. When processing each input, the routing mechanism activates only a subset of experts rather than all parameters, significantly reducing per-inference computation while maintaining the model's overall capacity. MoE is widely applied in large-scale language models and deep learning systems as a key technique for balancing model scale and computational efficiency.
Core Facts
NVIDIA Nemotron 3.5 Super VL is a 120B-parameter MoE model with 12B active parameters, supporting up to a 1 million token context, aimed at document understanding, computer use and agent scenarios.
Flash models typically use knowledge distillation technology, training smaller-parameter, faster-inference student models with flagship models as the teacher model.
The theoretical roots of the Mixture of Experts model can be traced back to the mixture of experts system proposed by Jacobs et al. in 1991.
Timeline (last 90 days)
The cost of the MoE architecture is that VRAM usage is still calculated based on total parameters, so the deployment threshold does not drop proportionally.
A larger number of experts means greater total capacity and broader knowledge coverage, but it brings deployment challenges such as memory footprint, load balancing, and communication overhead.
Some users hoped the next-generation model would "push N-grams to the limit" and expected Qwen4 Flash Next to have a larger N-gram table or more experts.
MoE models need to load weights according to total parameter count and are more sensitive to memory bandwidth and batching strategies.
The MoE architecture splits the feed-forward network (FFN) layers in the Transformer into multiple parallel experts, and each token is routed to only a few experts for computation.
The MoE architecture breaks the linear relationship in dense models where inference cost is proportional to parameter count, allowing parameter counts to keep growing while per-token computation remains within a controllable range.
The memory footprint of an MoE model does not decline proportionally due to sparse activation; all expert parameters still need to reside in memory.
"A3B" in the name refers to the mixture-of-experts (MoE) architecture, where total parameters are large but only some experts are activated per token, so VRAM usage and speed during inference are close to those of a smaller dense model.
Grouped GEMM packs multiple matrix multiplications of different shapes into a single kernel call, typically used for parallel computation of different expert weights in MoE (mixture-of-experts) models, and can significantly reduce kernel launch overhead.
Each Transformer layer of DeepSeek's MoE model contains dozens or even hundreds of expert feed-forward networks, and each token is assigned by the router to only a few experts.
40 more timeline events
All Facts (20)
NVIDIA Nemotron 3.5 Super VL is a 120B-parameter MoE model with 12B active parameters, supporting up to a 1 million token context, aimed at document understanding, computer use and agent scenarios.
90%VerifiedFlash models typically use knowledge distillation technology, training smaller-parameter, faster-inference student models with flagship models as the teacher model.
90%VerifiedThe theoretical roots of the Mixture of Experts model can be traced back to the mixture of experts system proposed by Jacobs et al. in 1991.
90%VerifiedDeepSeek-V3 adopts a Mixture of Experts (MoE) architecture with a total parameter count of 671 billion, but only approximately 37 billion parameters are activated per inference.
90%VerifiedGoogle's Switch Transformer and Mistral's Mixtral 8x7B are typical representatives of the MoE architecture.
90%VerifiedThe MoE architecture typically activates only 10%–20% of total parameters during each inference
90%VerifiedThe MoE architecture divides a model's internals into multiple expert sub-networks, with a gating network dynamically selecting a small number of experts to participate in computation during each inference, so that the actual number of activated parameters is far smaller than the total parameter count.
90%VerifiedDeepSeek is a series of large language models developed by the Chinese artificial intelligence company Deepseek.
90%VerifiedThe VRAM usage and inference latency of MoE architectures are calculated based on the number of activated parameters, not the total parameter count.
90%VerifiedMuse Spark Open uses an MoE architecture, with 816B total parameters and 42B activated parameters.
85%VerifiedQwen3.8-Flash-Next adopts a Mixture of Experts (MoE) architecture with 125B total parameters and 6B activated parameters.
85%VerifiedThe large-capacity, small-activation design of MoE allows a model's knowledge capacity and inference cost to be decoupled.
85%VerifiedMixture of Experts (MoE) models dynamically select a small number of experts to activate for each input token through a gating mechanism, thereby reducing the computational cost per inference while maintaining model capability.
85%VerifiedThe Mixture-of-Experts (MoE) architecture replaces feed-forward network layers with multiple expert sub-networks and dynamically activates a subset of experts via a router, so that the number of actually activated parameters is only a small fraction of the total.
85%VerifiedGoogle's Gemini 1.5 series has been confirmed to use a MoE (Mixture of Experts) architecture.
85%VerifiedGPT-4和Mixtral等主流大模型也采用了MoE架构
85%VerifiedQwen 3.8 Flash Next adopts a Mixture of Experts (MoE) architecture.
80%VerifiedDeepSeek V4 Flash is an MoE multimodal model with a total parameter count of 284B.
80%VerifiedDeepSeek V4 Flash is a cost-effective inference model released by DeepSeek, optimized for speed and cost.
80%VerifiedThe MoE architecture uses multiple expert networks and a routing mechanism to selectively activate only a subset of experts during each forward pass, reducing computational overhead while maintaining large model capacity.
80%