Mamba-2
Mamba 架构的改进版本，通过结构化状态空间对偶（SSD）进一步优化了状态空间模型的计算效率和硬件利用率。
Timeline (last 90 days)
The fusion of Mamba-style state space models with Transformers is moving from papers to usable open-source weights.
Nemotron 3.5 Lightning uses a hybrid structure that interleaves Mamba-2 with MoE layers, paired with a small number of attention layers.
NVIDIA's Nemotron-H replaces almost all attention layers with Mamba-2 layers.
Falcon-H1 runs attention heads and Mamba-2 heads in parallel within the same block, rather than replacing them layer by layer.
Most SSM and linear attention models use a linear attention mechanism at the bottom layer, with Mamba-2 implemented through the SSD structure and xLSTM implemented through covariance updates.
Zamba2 (Zyphra) is designed as a Mamba-2 backbone plus shared-weight attention blocks inserted every N layers.
The SSD (State Space Duality) framework proposed by Mamba-2 reveals that SSMs with a specific structure are mathematically equivalent to a masked form of linear attention, allowing training to be parallelized using matrix multiplication.
SSM 是一类以线性递推方式处理序列的架构，代表如 Mamba、Mamba-2 等。
Mamba-2 and hybrid architectures combining Mamba with Transformers (such as Jamba) have been released, alternately stacking attention mechanisms and SSM layers.
All Facts (9)
Nemotron 3.5 Lightning uses a hybrid structure that interleaves Mamba-2 with MoE layers, paired with a small number of attention layers.
50%UnverifiedThe SSD (State Space Duality) framework proposed by Mamba-2 reveals that SSMs with a specific structure are mathematically equivalent to a masked form of linear attention, allowing training to be parallelized using matrix multiplication.
50%UnverifiedNVIDIA's Nemotron-H replaces almost all attention layers with Mamba-2 layers.
50%UnverifiedMost SSM and linear attention models use a linear attention mechanism at the bottom layer, with Mamba-2 implemented through the SSD structure and xLSTM implemented through covariance updates.
50%UnverifiedZamba2 (Zyphra) is designed as a Mamba-2 backbone plus shared-weight attention blocks inserted every N layers.
50%UnverifiedFalcon-H1 runs attention heads and Mamba-2 heads in parallel within the same block, rather than replacing them layer by layer.
50%UnverifiedSSM 是一类以线性递推方式处理序列的架构，代表如 Mamba、Mamba-2 等。
50%UnverifiedThe fusion of Mamba-style state space models with Transformers is moving from papers to usable open-source weights.
50%UnverifiedMamba-2 and hybrid architectures combining Mamba with Transformers (such as Jamba) have been released, alternately stacking attention mechanisms and SSM layers.
50%