Jamba
A hybrid-architecture large language model developed by AI21 Labs that alternately stacks Transformer attention layers with Mamba SSM layers and incorporates a Mixture-of-Experts structure. The design balances long-context capability with inference efficiency.
Timeline (last 90 days)
MiniMax-01 and Jamba push the ratio of linear attention to softmax attention to 7:1 or 8:1, and additionally stack an MoE structure on top.
Inception Labs' diffusion model achieves tokens/second on coding tasks far exceeding any Transformer or even hybrid architectures such as Jamba.
Mamba-2 and hybrid architectures combining Mamba with Transformers (such as Jamba) have been released, alternately stacking attention mechanisms and SSM layers.
The hybrid attention architecture concept is also reflected in models such as Jamba (Mamba + Attention hybrid) and Zamba.
AI21 Labs' Jamba alternates Mamba layers and attention layers within a 52B-parameter MoE architecture, making it one of the earliest large-scale commercial hybrid models.
2024年涌现了多个线性或近线性复杂度序列模型，包括DeepMind的Griffin、AI21 Labs的Jamba、HGRN2和Stanford的Based
All Facts (6)
AI21 Labs' Jamba alternates Mamba layers and attention layers within a 52B-parameter MoE architecture, making it one of the earliest large-scale commercial hybrid models.
70%UnverifiedMiniMax-01 and Jamba push the ratio of linear attention to softmax attention to 7:1 or 8:1, and additionally stack an MoE structure on top.
50%UnverifiedInception Labs' diffusion model achieves tokens/second on coding tasks far exceeding any Transformer or even hybrid architectures such as Jamba.
50%UnverifiedThe hybrid attention architecture concept is also reflected in models such as Jamba (Mamba + Attention hybrid) and Zamba.
50%Unverified2024年涌现了多个线性或近线性复杂度序列模型，包括DeepMind的Griffin、AI21 Labs的Jamba、HGRN2和Stanford的Based
50%UnverifiedMamba-2 and hybrid architectures combining Mamba with Transformers (such as Jamba) have been released, alternately stacking attention mechanisms and SSM layers.
50%