Expert Parallelism
A distributed training parallelism strategy for MoE models in which different expert sub-networks are assigned to different GPU devices. Token routing and aggregation are achieved through All-to-All communication, enabling efficient training of extremely large-scale MoE models.
Core Facts
Timeline (last 90 days)
In expert parallelism, after routing decisions are made, an All-to-All communication is required to send tokens to the corresponding devices and then aggregate the results
In MoE models, each token is routed to only a small number of experts, producing two communication primitives: Dispatch and Combine
The essence of an expert agent is to build a digital avatar of a domain expert.
All Facts (3)
The essence of an expert agent is to build a digital avatar of a domain expert.
75%UnverifiedIn expert parallelism, after routing decisions are made, an All-to-All communication is required to send tokens to the corresponding devices and then aggregate the results
50%UnverifiedIn MoE models, each token is routed to only a small number of experts, producing two communication primitives: Dispatch and Combine
50%