Gated DeltaNet
A recurrent model architecture that combines gating mechanisms with Delta-rule linear attention layers. It belongs to the family of linear attention hybrid designs and incurs low memory and compute overhead when processing long contexts.
Timeline (last 90 days)
Mamba-2, Gated DeltaNet and KDA are three new state update mechanisms used to replace some attention layers, achieving efficient state memory through a recursive structure with linear complexity.
Gated DeltaNet starts from DeltaNet and adds a gating mechanism that dynamically decides when to retain historical information and when to overwrite the old state, performing more stably on tasks that require precise recall.
NoteBank is a sparse episodic memory module that can be attached directly to pretrained recurrent models and hybrid language models, such as Gated DeltaNet (GDN) and Mamba-2, without changing the backbone network's weights or architecture, and without needing external embedding models, a separately trained retriever, or an independent reader network.
All Facts (3)
Mamba-2, Gated DeltaNet and KDA are three new state update mechanisms used to replace some attention layers, achieving efficient state memory through a recursive structure with linear complexity.
50%UnverifiedGated DeltaNet starts from DeltaNet and adds a gating mechanism that dynamically decides when to retain historical information and when to overwrite the old state, performing more stably on tasks that require precise recall.
50%UnverifiedNoteBank is a sparse episodic memory module that can be attached directly to pretrained recurrent models and hybrid language models, such as Gated DeltaNet (GDN) and Mamba-2, without changing the backbone network's weights or architecture, and without needing external embedding models, a separately trained retriever, or an independent reader network.
50%