Mamba-2
An improved version of the Mamba architecture that further optimizes the computational efficiency and hardware utilization of state space models through structured state space duality (SSD).
Timeline (last 90 days)
Nemotron 3.5 Super VL 采用 Mamba-2/Transformer 混合 MoE 架构，每个 MoE 层有 512 个路由专家，视觉侧搭配 C-RADIO 编码器，支持文本、JPEG/PNG 图像及最长 2 分钟 MP4 视频，上下文上限 1M token，并原生支持多 token 预测。
Mamba-2 builds on selective state space models by introducing a more structured state transition matrix, expressing the state update in matrix multiplication form, thereby achieving higher throughput during training.
Mamba-2, Gated DeltaNet and KDA are three new state update mechanisms used to replace some attention layers, achieving efficient state memory through a recursive structure with linear complexity.
NoteBank is a sparse episodic memory module that can be attached directly to pretrained recurrent models and hybrid language models, such as Gated DeltaNet (GDN) and Mamba-2, without changing the backbone network's weights or architecture, and without needing external embedding models, a separately trained retriever, or an independent reader network.
Nemotron 3.5 Bolt's architecture adopts LatentMoE mixing Mamba-2 and Transformer, and the vision side uses the C-RADIO v4-H encoder.
Nemotron 3.5 Lightning uses a hybrid structure that interleaves Mamba-2 with MoE layers, paired with a small number of attention layers.
The fusion of Mamba-style state space models with Transformers is moving from papers to usable open-source weights.
Zamba2 (Zyphra) is designed as a Mamba-2 backbone plus shared-weight attention blocks inserted every N layers.
The SSD (State Space Duality) framework proposed by Mamba-2 reveals that SSMs with a specific structure are mathematically equivalent to a masked form of linear attention, allowing training to be parallelized using matrix multiplication.
Most SSM and linear attention models use a linear attention mechanism at the bottom layer, with Mamba-2 implemented through the SSD structure and xLSTM implemented through covariance updates.
4 more timeline events
All Facts (14)
Nemotron 3.5 Super VL 采用 Mamba-2/Transformer 混合 MoE 架构，每个 MoE 层有 512 个路由专家，视觉侧搭配 C-RADIO 编码器，支持文本、JPEG/PNG 图像及最长 2 分钟 MP4 视频，上下文上限 1M token，并原生支持多 token 预测。
50%UnverifiedMamba-2 builds on selective state space models by introducing a more structured state transition matrix, expressing the state update in matrix multiplication form, thereby achieving higher throughput during training.
50%UnverifiedMamba-2, Gated DeltaNet and KDA are three new state update mechanisms used to replace some attention layers, achieving efficient state memory through a recursive structure with linear complexity.
50%UnverifiedNoteBank is a sparse episodic memory module that can be attached directly to pretrained recurrent models and hybrid language models, such as Gated DeltaNet (GDN) and Mamba-2, without changing the backbone network's weights or architecture, and without needing external embedding models, a separately trained retriever, or an independent reader network.
50%UnverifiedNemotron 3.5 Bolt's architecture adopts LatentMoE mixing Mamba-2 and Transformer, and the vision side uses the C-RADIO v4-H encoder.
50%UnverifiedNemotron 3.5 Lightning uses a hybrid structure that interleaves Mamba-2 with MoE layers, paired with a small number of attention layers.
50%UnverifiedMost SSM and linear attention models use a linear attention mechanism at the bottom layer, with Mamba-2 implemented through the SSD structure and xLSTM implemented through covariance updates.
50%UnverifiedZamba2 (Zyphra) is designed as a Mamba-2 backbone plus shared-weight attention blocks inserted every N layers.
50%UnverifiedNVIDIA's Nemotron-H replaces almost all attention layers with Mamba-2 layers.
50%UnverifiedThe SSD (State Space Duality) framework proposed by Mamba-2 reveals that SSMs with a specific structure are mathematically equivalent to a masked form of linear attention, allowing training to be parallelized using matrix multiplication.
50%UnverifiedFalcon-H1 runs attention heads and Mamba-2 heads in parallel within the same block, rather than replacing them layer by layer.
50%UnverifiedSSM 是一类以线性递推方式处理序列的架构，代表如 Mamba、Mamba-2 等。
50%UnverifiedThe fusion of Mamba-style state space models with Transformers is moving from papers to usable open-source weights.
50%UnverifiedMamba-2 and hybrid architectures combining Mamba with Transformers (such as Jamba) have been released, alternately stacking attention mechanisms and SSM layers.
50%