MoE
Mixture of Experts (MoE) is a machine learning architectural paradigm consisting of multiple specialized sub-networks called "experts" and a gating and routing mechanism. When processing each input, the routing mechanism activates only a subset of experts rather than all parameters, significantly reducing per-inference computation while maintaining the model's overall capacity. MoE is widely applied in large-scale language models and deep learning systems as a key technique for balancing model scale and computational efficiency.
Core Facts
NVIDIA Nemotron 3.5 Super VL is a 120B-parameter MoE model with 12B active parameters, supporting up to a 1 million token context, aimed at document understanding, computer use and agent scenarios.
Flash models typically use knowledge distillation technology, training smaller-parameter, faster-inference student models with flagship models as the teacher model.
GPT-4 uses a Mixture of Experts (MoE) architecture, with an estimated total parameter count of 1.8 trillion, and approximately 1/8 of the total parameters are actually activated during each inference.
Timeline (last 90 days)
When running MoE models locally, expert offloading is a key technique that allows infrequently used expert parameters to be stored in external storage (such as CPU memory or SSD) and loaded to the GPU when needed.
All the parameters of a single MoE layer may be as large as tens or even hundreds of GB, but only a small fraction is actually used in a single forward pass.
Under expert parallelism, each MoE layer goes through three steps: all-to-all token dispatch, grouped GEMM, and combine collection.
Engine-hardware co-optimization can leverage Tensor Core sparse computation support or memory hierarchy characteristics to prefetch expert weights during the routing stage and overlap computation with data transfer.
Load balancing optimizations at the MoE scheduling level include: grouping requests by expert affinity, dynamically adjusting batch size, and using a capacity factor to limit the number of tokens per expert and handle overflow.
NVIDIA's optimization does not change the routing logic or introduce computational approximations, but rather improves execution efficiency, which explains why no change in accuracy was detected.
Common memory access optimizations at the MoE kernel level include: grouping expert weights by access frequency, using contiguous memory layouts to reduce fragmentation, and fusing routing and expert computation into a single kernel to reduce write-back of intermediate results.
The bottlenecks of MoE inference lie mainly in memory access and scheduling: expert weights are stored in a dispersed manner, dynamic routing results lead to irregular memory access patterns, and load within a batch is unbalanced.
Sonar-MoE proposes directly updating expert tensors during backpropagation and immediately releasing the gradients after the update, thereby reducing peak memory by 19% to 22%.
The 3-bit and NVFP4 versions of Qwen3.8-27B scored 0.39 and 0.34 respectively, trailing the MoE-architecture Flash-Next, with a paired difference of +0.16, P = 0.92.
40 more timeline events
All Facts (20)
NVIDIA Nemotron 3.5 Super VL is a 120B-parameter MoE model with 12B active parameters, supporting up to a 1 million token context, aimed at document understanding, computer use and agent scenarios.
90%VerifiedFlash models typically use knowledge distillation technology, training smaller-parameter, faster-inference student models with flagship models as the teacher model.
90%VerifiedGPT-4 uses a Mixture of Experts (MoE) architecture, with an estimated total parameter count of 1.8 trillion, and approximately 1/8 of the total parameters are actually activated during each inference.
90%VerifiedThe theoretical roots of the Mixture of Experts model can be traced back to the mixture of experts system proposed by Jacobs et al. in 1991.
90%VerifiedDeepSeek-V3 adopts a Mixture of Experts (MoE) architecture with a total parameter count of 671 billion, but only approximately 37 billion parameters are activated per inference.
90%VerifiedGoogle's Switch Transformer and Mistral's Mixtral 8x7B are typical representatives of the MoE architecture.
90%VerifiedThe MoE architecture typically activates only 10%–20% of total parameters during each inference
90%VerifiedThe MoE architecture divides a model's internals into multiple expert sub-networks, with a gating network dynamically selecting a small number of experts to participate in computation during each inference, so that the actual number of activated parameters is far smaller than the total parameter count.
90%VerifiedDeepSeek is a series of large language models developed by the Chinese artificial intelligence company Deepseek.
90%VerifiedThe VRAM usage and inference latency of MoE architectures are calculated based on the number of activated parameters, not the total parameter count.
90%VerifiedMuse Spark Open uses an MoE architecture, with 816B total parameters and 42B activated parameters.
85%VerifiedMistral Large 4 adopts a sparse Mixture of Experts (MoE) architecture, with 49 billion (49B) parameters actually activated during each inference.
85%VerifiedQwen3.8-Flash-Next adopts a Mixture of Experts (MoE) architecture with 125B total parameters and 6B activated parameters.
85%VerifiedThe large-capacity, small-activation design of MoE allows a model's knowledge capacity and inference cost to be decoupled.
85%VerifiedMixture of Experts (MoE) models dynamically select a small number of experts to activate for each input token through a gating mechanism, thereby reducing the computational cost per inference while maintaining model capability.
85%VerifiedThe Mixture-of-Experts (MoE) architecture replaces feed-forward network layers with multiple expert sub-networks and dynamically activates a subset of experts via a router, so that the number of actually activated parameters is only a small fraction of the total.
85%VerifiedGoogle's Gemini 1.5 series has been confirmed to use a MoE (Mixture of Experts) architecture.
85%VerifiedGPT-4和Mixtral等主流大模型也采用了MoE架构
85%VerifiedQwen 3.8 Flash Next adopts a Mixture of Experts (MoE) architecture.
80%VerifiedDeepSeek V4 Flash is an MoE multimodal model with a total parameter count of 284B.
80%