Gated Delta Net
A linear attention variant that replaces the traditional KV cache with an updatable fixed-size state matrix. It compresses historical information through gating and write strength, supporting constant memory for long sequences.
Core Facts
Timeline (last 90 days)
Gated Delta Net is a linear attention variant that uses an updatable state matrix instead of the key-value cache in traditional attention, with constant memory usage on long sequences, but the compression process is inherently lossy.
A certain version of Qwen 3.5 has 32 layers in total, and three of every four layers use the Gated Delta Net structure.
Setting the gating of the Gated Delta Net state matrix to 1, the write strength to 1, and making each key align with only one row enables each write to replace an entire row and read values back precisely by row number.
In the author's small-model validation, the value 4 written into a state matrix with gating 0.9 decayed to 1.55 after 9 steps; with a write strength of 0.5, the value 7 was only half written; and when keys did not align with a single row, a large amount of noise appeared.
In the 8-bit CPU validation, changing the gate of the register state matrix from 1 to 0.999 makes the same program output 132; when the gate is 0.99 the output is 61; when the gate is 0.999 errors begin at the 31st token.
All Facts (7)
Qwen3.5 uses a hybrid attention architecture that mixes Gated Delta Net (a linear attention variant) with Gated Attention (standard attention).
65%UnverifiedGated Delta Net将状态空间复杂度从标准注意力的O(N²)降到O(N)量级
90%UnverifiedGated Delta Net is a linear attention variant that uses an updatable state matrix instead of the key-value cache in traditional attention, with constant memory usage on long sequences, but the compression process is inherently lossy.
50%UnverifiedA certain version of Qwen 3.5 has 32 layers in total, and three of every four layers use the Gated Delta Net structure.
50%UnverifiedSetting the gating of the Gated Delta Net state matrix to 1, the write strength to 1, and making each key align with only one row enables each write to replace an entire row and read values back precisely by row number.
50%UnverifiedIn the author's small-model validation, the value 4 written into a state matrix with gating 0.9 decayed to 1.55 after 9 steps; with a write strength of 0.5, the value 7 was only half written; and when keys did not align with a single row, a large amount of noise appeared.
50%UnverifiedIn the 8-bit CPU validation, changing the gate of the register state matrix from 1 to 0.999 makes the same program output 132; when the gate is 0.99 the output is 61; when the gate is 0.999 errors begin at the 31st token.
50%