DeltaNet
A linear attention variant architecture that replaces softmax attention with a delta rule-based incremental update mechanism, reducing sequence-processing complexity from O(n²) to O(n). The design is suited for efficient handling of long-context inputs while maintaining competitive modeling capacity.
Core Facts
Timeline (last 90 days)
KronLSTM is competitive with existing linear attention architectures and often surpasses models including the (Gated) DeltaNet family, especially on long-context benchmarks.
The DeltaNet series uses the Delta Rule (similar to error correction in online learning) for targeted writing of associations.
DeltaNet belongs to linear attention/state space models, replacing the full-history KV storage in standard attention with an updatable recurrent state matrix, so computation and storage grow linearly rather than quadratically with sequence length.
Gated DeltaNet starts from DeltaNet and adds a gating mechanism that dynamically decides when to retain historical information and when to overwrite the old state, performing more stably on tasks that require precise recall.
The Qwen3.8 Flash Next model has 48 layers in total, constructed by repeating "3 gated DeltaNet layers + 1 sparse injection" 12 times, containing approximately 245,000 expert instances in total, with traditional attention cache retained in only 12 layers.
DeltaNet is a variant of linear attention that replaces softmax attention with a delta update rule, reducing sequence processing complexity from O(n²) to O(n).
DeltaNet's incremental update mechanism keeps GPU memory usage from growing with sequence length, maintaining a constant size instead, because it maintains a fixed-size hidden state rather than a full KV Cache.
DeltaNet belongs to the linear attention mechanism family, designed to reduce the computational complexity of attention from O(n²) to O(n).
All Facts (9)
Qwen3.5 uses a hybrid attention architecture that mixes Gated Delta Net (a linear attention variant) with Gated Attention (standard attention).
65%UnverifiedKronLSTM is competitive with existing linear attention architectures and often surpasses models including the (Gated) DeltaNet family, especially on long-context benchmarks.
50%UnverifiedThe DeltaNet series uses the Delta Rule (similar to error correction in online learning) for targeted writing of associations.
50%UnverifiedDeltaNet belongs to linear attention/state space models, replacing the full-history KV storage in standard attention with an updatable recurrent state matrix, so computation and storage grow linearly rather than quadratically with sequence length.
50%UnverifiedGated DeltaNet starts from DeltaNet and adds a gating mechanism that dynamically decides when to retain historical information and when to overwrite the old state, performing more stably on tasks that require precise recall.
50%UnverifiedThe Qwen3.8 Flash Next model has 48 layers in total, constructed by repeating "3 gated DeltaNet layers + 1 sparse injection" 12 times, containing approximately 245,000 expert instances in total, with traditional attention cache retained in only 12 layers.
50%UnverifiedDeltaNet is a variant of linear attention that replaces softmax attention with a delta update rule, reducing sequence processing complexity from O(n²) to O(n).
50%UnverifiedDeltaNet's incremental update mechanism keeps GPU memory usage from growing with sequence length, maintaining a constant size instead, because it maintains a fixed-size hidden state rather than a full KV Cache.
50%UnverifiedDeltaNet belongs to the linear attention mechanism family, designed to reduce the computational complexity of attention from O(n²) to O(n).
50%