Sparse MoE
A mixture-of-experts model architecture in which only a subset of expert subnetworks is activated for each token, decoupling the total parameter scale from the computation cost per pass. It is commonly used in training trillion-scale large models.
Timeline (last 90 days)
The sparse MoE architecture keeps the computation cost roughly constant while expanding the total number of parameters, but storage overhead does not decrease.
The sparse MoE architecture faces challenges such as load imbalance and communication overhead.
The sparse MoE architecture is becoming a mainstream choice for large models.
Step 5 Preview is StepFun's newly released next-generation flagship foundation model, using a sparse MoE architecture with a total parameter count of 600 billion, activating approximately 27 billion parameters per token.
The sparse MoE architecture dynamically selects a small number of expert sub-networks to participate in computation through a gating mechanism, rather than activating all parameters.
All Facts (5)
The sparse MoE architecture keeps the computation cost roughly constant while expanding the total number of parameters, but storage overhead does not decrease.
50%UnverifiedThe sparse MoE architecture faces challenges such as load imbalance and communication overhead.
50%UnverifiedThe sparse MoE architecture is becoming a mainstream choice for large models.
50%UnverifiedStep 5 Preview is StepFun's newly released next-generation flagship foundation model, using a sparse MoE architecture with a total parameter count of 600 billion, activating approximately 27 billion parameters per token.
50%UnverifiedThe sparse MoE architecture dynamically selects a small number of expert sub-networks to participate in computation through a gating mechanism, rather than activating all parameters.
50%