Mixture-of-Transformers
A hybrid Transformer architecture that runs multiple expert types in parallel, such as autoregressive and non-autoregressive experts. They share attention computation but use separate normalization, projection, and feed-forward layers.
Timeline (last 90 days)
Mixture-of-Transformers (MoT) is a Transformer architecture variant that, on the basis of shared backbone weights, configures independent attention or normalization adaptation modules for different modalities or tasks.
Ask4NV is a novel view generation method built on a pretrained unified multimodal model, adopting a Mixture-of-Transformers (MoT) architecture.
Mixture-of-Transformers (MoT) keeps an independent Transformer branch for each modality, allowing information interaction through cross-modal attention at specified layers.
YuE2 underlyingly uses a 28-layer Mixture-of-Transformers (MoT) architecture, running autoregressive (AR) and non-autoregressive (NAR) expert types in parallel, which share attention computation but use separate normalization, projection, and feed-forward layers.
All Facts (4)
Mixture-of-Transformers (MoT) is a Transformer architecture variant that, on the basis of shared backbone weights, configures independent attention or normalization adaptation modules for different modalities or tasks.
50%UnverifiedAsk4NV is a novel view generation method built on a pretrained unified multimodal model, adopting a Mixture-of-Transformers (MoT) architecture.
50%UnverifiedMixture-of-Transformers (MoT) keeps an independent Transformer branch for each modality, allowing information interaction through cross-modal attention at specified layers.
50%UnverifiedYuE2 underlyingly uses a 28-layer Mixture-of-Transformers (MoT) architecture, running autoregressive (AR) and non-autoregressive (NAR) expert types in parallel, which share attention computation but use separate normalization, projection, and feed-forward layers.
50%