Model

Nemotron-H

A hybrid-architecture model introduced by NVIDIA that replaces nearly all of its attention layers with Mamba-2 layers. By retaining only a small number of attention layers while relying predominantly on state-space sequence modeling, the design aims to reduce the computational cost of long-context processing.

Timeline (last 90 days)

Oct 16

The Nemotron-H decoder uses 512-expert MoE layers, and at inference time dynamic routing activates only 22 experts, so of the 120B total parameters only 12B participate in computation per step.

Unverified50%

Oct 14

NVIDIA's Nemotron-H replaces almost all attention layers with Mamba-2 layers.

Unverified50%

All Facts (2)

Unverified

The Nemotron-H decoder uses 512-expert MoE layers, and at inference time dynamic routing activates only 22 experts, so of the 120B total parameters only 12B participate in computation per step.

50%Unverified

NVIDIA's Nemotron-H replaces almost all attention layers with Mamba-2 layers.

50%

Source Articles

The Post-Transformer Landscape: Comparing Mainstream Architectures in 2026Oct 14