GShard
A large-scale distributed training model based on the MoE architecture proposed by Google. It uses expert sharding and automatic sharding techniques to enable efficient training of models with ultra-large-scale parameters.
Timeline (last 90 days)
The idea of MoE sparse activation was first validated in natural language processing by Switch Transformer, GShard, and others.
Entropy regularization techniques have been widely applied in the MoE domain, such as the load balancing loss in GShard and Switch Transformer.
Google has deep research experience in the MoE field, from Switch Transformer to GShard to the Gemini series.
2020年的GShard论文提出了辅助损失函数来惩罚不均衡的路由分配
All Facts (4)
The idea of MoE sparse activation was first validated in natural language processing by Switch Transformer, GShard, and others.
50%UnverifiedEntropy regularization techniques have been widely applied in the MoE domain, such as the load balancing loss in GShard and Switch Transformer.
50%UnverifiedGoogle has deep research experience in the MoE field, from Switch Transformer to GShard to the Gemini series.
50%Unverified2020年的GShard论文提出了辅助损失函数来惩罚不均衡的路由分配
50%