Google's Switch Transformer demonstrated in 2021 that routing each token to only a single expert per layer (Top-1 routing) within a Transformer can achieve stable training, improving pre-training speed by more than 7× compared to a dense model of equivalent scale.