[KongchangAI]
ConceptSparse MoE / 稀疏混合专家

Sparse MoE

A mixture-of-experts model architecture in which only a subset of expert subnetworks is activated for each token, decoupling the total parameter scale from the computation cost per pass. It is commonly used in training trillion-scale large models.

Timeline (last 90 days)

Oct 19

The sparse MoE architecture keeps the computation cost roughly constant while expanding the total number of parameters, but storage overhead does not decrease.

Unverified50%
Oct 19

The sparse MoE architecture faces challenges such as load imbalance and communication overhead.

Unverified50%
Oct 19

The sparse MoE architecture is becoming a mainstream choice for large models.

Unverified50%
Oct 12

Step 5 Preview is StepFun's newly released next-generation flagship foundation model, using a sparse MoE architecture with a total parameter count of 600 billion, activating approximately 27 billion parameters per token.

Unverified50%
Oct 12

The sparse MoE architecture dynamically selects a small number of expert sub-networks to participate in computation through a gating mechanism, rather than activating all parameters.

Unverified50%

All Facts (5)

Source Articles