DeltaNet
A linear attention variant architecture that replaces softmax attention with a delta rule-based incremental update mechanism, reducing sequence-processing complexity from O(n²) to O(n). The design is suited for efficient handling of long-context inputs while maintaining competitive modeling capacity.
Core Facts
Timeline (last 90 days)
The Qwen3.8 Flash Next model has 48 layers in total, constructed by repeating "3 gated DeltaNet layers + 1 sparse injection" 12 times, containing approximately 245,000 expert instances in total, with traditional attention cache retained in only 12 layers.
DeltaNet is a variant of linear attention that replaces softmax attention with a delta update rule, reducing sequence processing complexity from O(n²) to O(n).
DeltaNet's incremental update mechanism keeps GPU memory usage from growing with sequence length, maintaining a constant size instead, because it maintains a fixed-size hidden state rather than a full KV Cache.
DeltaNet belongs to the linear attention mechanism family, designed to reduce the computational complexity of attention from O(n²) to O(n).
All Facts (5)
Qwen3.5 uses a hybrid attention architecture that mixes Gated Delta Net (a linear attention variant) with Gated Attention (standard attention).
65%UnverifiedThe Qwen3.8 Flash Next model has 48 layers in total, constructed by repeating "3 gated DeltaNet layers + 1 sparse injection" 12 times, containing approximately 245,000 expert instances in total, with traditional attention cache retained in only 12 layers.
50%UnverifiedDeltaNet is a variant of linear attention that replaces softmax attention with a delta update rule, reducing sequence processing complexity from O(n²) to O(n).
50%UnverifiedDeltaNet's incremental update mechanism keeps GPU memory usage from growing with sequence length, maintaining a constant size instead, because it maintains a fixed-size hidden state rather than a full KV Cache.
50%UnverifiedDeltaNet belongs to the linear attention mechanism family, designed to reduce the computational complexity of attention from O(n²) to O(n).
50%