Switch Transformer
Switch Transformer is a large-scale language model proposed by Google based on the Mixture of Experts (MoE) architecture. Its core design replaces the feed-forward networks in standard Transformers with multiple expert networks, using a routing mechanism to dynamically activate a small subset of experts during each forward pass, thereby greatly scaling model parameter counts while maintaining computational efficiency. The model validated the effectiveness of MoE architectures in natural language processing and serves as an important reference for research on sparsely activated models.
Core Facts
Google's Switch Transformer, released in 2021, applied MoE at scale to Transformer language models, expanding model capacity to the trillion-parameter level.
Google's Switch Transformer and Mistral's Mixtral 8x7B are typical representatives of the MoE architecture.
The large-capacity, small-activation design of MoE allows a model's knowledge capacity and inference cost to be decoupled.
Timeline (last 90 days)
Entropy regularization techniques have been widely applied in the MoE domain, such as the load balancing loss in GShard and Switch Transformer.
Sparse activation routing mechanisms have been validated in research such as Switch Transformer and Mixtral, where input tokens are dynamically assigned to the most relevant expert sub-networks via a gating network.
Switch Transformer was released in 2021, and Mixtral 8×7B was released in 2023.
The core advantage of the MoE (Sparse Mixture of Experts) architecture is that it significantly reduces the computational cost of a single inference through sparse activation while maintaining a large parameter count; the architectural concept can be traced back to research by Jacobs et al. in 1991
The Mixture of Experts architecture can be traced back to the theoretical framework proposed by Jacobs et al. in 1991, but it was not truly introduced into the field of large-scale language models until Google's Switch Transformer paper in 2017.
Google has deep research experience in the MoE field, from Switch Transformer to GShard to the Gemini series.
Google's Switch Transformer and Mixtral 8x7B are classic representatives of the MoE architecture, and Qwen3, DeepSeek-V3, and others also adopt this architecture.
Mixture-of-Experts (MoE) architectures such as Google's Switch Transformer and Mistral AI's Mixtral have enormous total parameter counts but activate only a small fraction of them during each inference.
MoE架构可追溯到1991年Jacobs等人的工作，但直到2022年后才在大语言模型中大规模应用
Switch Transformer将每个Token的专家选择简化为Top-1，并引入辅助损失函数防止Token涌向少数热门专家
8 more timeline events
All Facts (20)
Google's Switch Transformer, released in 2021, applied MoE at scale to Transformer language models, expanding model capacity to the trillion-parameter level.
90%VerifiedGoogle's Switch Transformer and Mistral's Mixtral 8x7B are typical representatives of the MoE architecture.
90%VerifiedThe large-capacity, small-activation design of MoE allows a model's knowledge capacity and inference cost to be decoupled.
85%VerifiedThe Mixture-of-Experts (MoE) architecture replaces feed-forward network layers with multiple expert sub-networks and dynamically activates a subset of experts via a router, so that the number of actually activated parameters is only a small fraction of the total.
85%VerifiedThe core idea of MoE (Mixture of Experts) can be traced back to the early mixture of experts framework proposed by Jacobs et al. in 1991, and Google's Switch Transformer in 2021 brought it into the era of large-scale language models.
80%VerifiedThe MoE architecture was first proposed by Jacobs et al. in 1991 and in recent years has been brought back into the mainstream by models such as Google's Switch Transformer and Mixtral.
80%VerifiedMoE was reactivated in the era of large language models by Google's Switch Transformer (2021) and GLaM.
70%VerifiedGoogle validated the Mixture of Experts (MoE) architecture at scale in Switch Transformer.
70%VerifiedThe modern form of MoE can be traced back to a Google paper in 2017, and in 2022 Google's Switch Transformer demonstrated that trillion-parameter sparse models can be trained efficiently.
65%UnverifiedThe mixture-of-experts architecture can be traced back to research by Jacobs et al. in 1991; Google's Switch Transformer (2021) and Mistral's Mixtral 8x7B (2023) brought it back into the mainstream.
60%UnverifiedSwitch Transformer, proposed in 2021, simplified the number of experts routed to per token to 1.
60%UnverifiedGoogle's Switch Transformer and GLaM were the first to scale MoE to the trillion-parameter level.
60%UnverifiedEntropy regularization techniques have been widely applied in the MoE domain, such as the load balancing loss in GShard and Switch Transformer.
50%UnverifiedSparse activation routing mechanisms have been validated in research such as Switch Transformer and Mixtral, where input tokens are dynamically assigned to the most relevant expert sub-networks via a gating network.
50%UnverifiedThe core advantage of the MoE (Sparse Mixture of Experts) architecture is that it significantly reduces the computational cost of a single inference through sparse activation while maintaining a large parameter count; the architectural concept can be traced back to research by Jacobs et al. in 1991
50%UnverifiedThe Mixture of Experts architecture can be traced back to the theoretical framework proposed by Jacobs et al. in 1991, but it was not truly introduced into the field of large-scale language models until Google's Switch Transformer paper in 2017.
50%UnverifiedGoogle has deep research experience in the MoE field, from Switch Transformer to GShard to the Gemini series.
50%UnverifiedGoogle's Switch Transformer and Mixtral 8x7B are classic representatives of the MoE architecture, and Qwen3, DeepSeek-V3, and others also adopt this architecture.
50%UnverifiedMixture-of-Experts (MoE) architectures such as Google's Switch Transformer and Mistral AI's Mixtral have enormous total parameter counts but activate only a small fraction of them during each inference.
50%UnverifiedMoE架构可追溯到1991年Jacobs等人的工作，但直到2022年后才在大语言模型中大规模应用
50%