[KongchangAI]
ModelMamba 2

Mamba 2

The second-generation version of the state space model architecture, which introduces improvements such as a head structure and optimizes training and inference efficiency.

Timeline (last 90 days)

Nov 3

HybridMamba combines Mamba 1's per-coordinate decay, Mamba 2-style head structure, and Mamba 3's complex rotations and dual-endpoint writes, mixing real state blocks with rotated state blocks of different decay laws.

Unverified50%
Nov 3

The SISO configurations of Mamba 1-3 can be recovered as exact special cases of HybridMamba.

Unverified50%
Nov 3

Retrieval rank refers to the number of independent value channels that a model can write to and read from in parallel at a local interface, and is a key structural dimension distinguishing variants such as Mamba 1, Mamba 2, Mamba 3 and HybridMamba.

Unverified50%
Nov 3

On five large-scale H100 bfloat16 workloads, the authors' fused GPU kernels were roughly on par with Mamba 2 within the SSM core scope.

Expired50%
Nov 3

In small-model multi-token recall experiments, both HybridMamba and SSD-MIMO designs outperformed Mamba 1-3 at the hardest level.

Expired50%
Nov 3

On single-seed language models of about 500 million parameters and 2B tokens, HybridMamba and SSD-MIMO outperformed Mamba 1/2 on test loss, but the gains from routing did not translate into lower language model loss.

Expired50%

All Facts (3)

Source Articles