Unverified50% confidenceFactExact time
Google 的 Switch Transformer（2021）和 Mistral 的 Mixtral 8x7B（2023）都采用了稀疏 MoE 架构，即每次前向传播只激活少数几个专家
1
Sources
50%
Confidence
Long-term
Relevance
7/17/2026
First Seen
Sources
Related Claims
UnverifiedSwitch Transformer and the Mixtral series have widely adopted the MoE architecture.74% similarUnverifiedMeta's LLaMA series, Google's Gemma, Mistral AI's Mistral/Mixtral, and Alibaba's Qwen series are all supported in Transformers.67% similarUnverifiedGoogle 2017年提出的Sparsely-Gated MoE首次将稀疏门控引入Transformer规模的序列模型，可在不显著增加计算量前提下将模型容量扩大1000倍以上65% similarUnverifiedMainstream open-source models such as Meta's LLaMA series, Google's Gemma, Alibaba's Qwen, and DeepSeek all prioritize compatibility with the Transformers format.64% similarVerifiedThe MoE architecture replaces the feed-forward network layers of the Transformer with multiple parallel expert sub-networks, dynamically selecting 2–4 experts to activate for each token via a routing network.62% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/544377API
curl https://kongchang.com/api/v1/knowledge/claims/544377MCP
get_claim(id=544377)