Mixture of Transformers
An architecture that combines multiple heterogeneous Transformer submodules within a single model, each designed for a specific modality or task type. A routing mechanism dynamically invokes submodules based on input type and task instructions, enabling unified multi-task processing and avoiding negative transfer between tasks.
Timeline (last 90 days)
The MoT in SenseNova-Vision-7B-MoT refers to Mixture of Transformers, an architecture that combines different types of expert modules within a single model, enabling it to handle vision-language multi-task inputs and outputs.
MoT elevates the idea of MoE to the module level; the model internally contains multiple heterogeneous Transformer submodules, each designed for a specific modality or task type, with a routing mechanism deciding which submodules to invoke based on the input type and task instruction.
All Facts (2)
The MoT in SenseNova-Vision-7B-MoT refers to Mixture of Transformers, an architecture that combines different types of expert modules within a single model, enabling it to handle vision-language multi-task inputs and outputs.
50%UnverifiedMoT elevates the idea of MoE to the module level; the model internally contains multiple heterogeneous Transformer submodules, each designed for a specific modality or task type, with a routing mechanism deciding which submodules to invoke based on the input type and task instruction.
50%