GDN
A recurrent state model architecture based on linear attention. It introduces a gating mechanism to control the forgetting rate of historical information and uses the Delta rule to make state updates approximate online gradient steps.
Timeline (last 90 days)
The study proposes two methods, global gradient-aware allocation (GGA) and pure-write sliding window (PW), to simultaneously address the challenges of precision allocation and recent-information maintenance within a storage budget.
GDN (Gated DeltaNet) introduces a gating mechanism and the Delta rule on top of linear attention; the gating controls the forgetting rate of historical information, and the Delta rule makes the state update approximate a gradient step in online learning.
In GDN, quantization residuals do not merely exist as noise but are amplified or persisted through the gating and Delta updates, thereby distorting all subsequent readouts, so their harm is greater than that of ordinary accumulators.
PW independently maintains the pure-write content of the most recent period, stores it as FP8 vector pairs that fully participate in the GDN state transition, and merges it into the low-bit historical state only after it leaves the sliding window.
On six RULER needle-in-a-haystack tasks at a one-million-token setting, the paper reports that GDN's average accuracy improved from 1.7% to 70.3%.
All Facts (5)
The study proposes two methods, global gradient-aware allocation (GGA) and pure-write sliding window (PW), to simultaneously address the challenges of precision allocation and recent-information maintenance within a storage budget.
50%UnverifiedGDN (Gated DeltaNet) introduces a gating mechanism and the Delta rule on top of linear attention; the gating controls the forgetting rate of historical information, and the Delta rule makes the state update approximate a gradient step in online learning.
50%UnverifiedIn GDN, quantization residuals do not merely exist as noise but are amplified or persisted through the gating and Delta updates, thereby distorting all subsequent readouts, so their harm is greater than that of ordinary accumulators.
50%UnverifiedPW independently maintains the pure-write content of the most recent period, stores it as FP8 vector pairs that fully participate in the GDN state transition, and merges it into the low-bit historical state only after it leaves the sliding window.
50%UnverifiedOn six RULER needle-in-a-haystack tasks at a one-million-token setting, the paper reports that GDN's average accuracy improved from 1.7% to 70.3%.
50%