Linear Attention
A simplified variant of standard Transformer attention that replaces the softmax exponential kernel with a linear kernel. Its forward computation can be expressed in closed form as matrix multiplications, enabling rigorous asymptotic analysis with classical statistical learning tools. It is commonly used to study in-context learning mechanisms.
Timeline (last 90 days)
Different types of attention mechanisms (such as softmax attention, linear attention, etc.) belong to the same family in the statistical mechanics sense and their phase structure can be analyzed with a unified framework.
GLM-5.3-Flash 的注意力架构将计算拆为线性注意力（O(n)递归状态，处理近期token）和稀疏注意力（轻量索引器回扫全局上下文，只拉取相关远端片段）两条路径。
Linear attention methods (such as Performer and RWKV) compress complexity to O(N) via kernel function approximation, but their precise recall capability is typically significantly weaker than full attention.
Many linear attention or sparse attention methods lose the ability to precisely retrieve historical information while compressing computation.
Gated Linear Attention (GLA) reduces computational complexity from O(N²) to O(N) through a linearized attention mechanism.
Minimax has trained a 456B-parameter MoE using a linear attention approach, but still requires occasional use of full attention.
Linear attention, sparse attention, and State Space Models (such as Mamba) can reduce computational complexity to linear levels, opening the door to million-token context windows.
DeltaNet's incremental update mechanism keeps GPU memory usage from growing with sequence length, maintaining a constant size instead, because it maintains a fixed-size hidden state rather than a full KV Cache.
Hybrid architectures (linear attention + standard attention) are more efficient than pure full attention on long-context tasks, but linear attention may be slightly inferior to standard attention on certain reasoning tasks that require precise global dependencies
Qwen3.8 Flash Next adopts a hybrid architecture whose core technology is Gated DeltaNet, combining linear attention with standard full attention
4 more timeline events
All Facts (14)
Qwen3 27B combines three linear attention layers with one full attention layer in a recurrent structure at a ratio of 3:1.
60%UnverifiedDifferent types of attention mechanisms (such as softmax attention, linear attention, etc.) belong to the same family in the statistical mechanics sense and their phase structure can be analyzed with a unified framework.
50%UnverifiedGLM-5.3-Flash 的注意力架构将计算拆为线性注意力（O(n)递归状态，处理近期token）和稀疏注意力（轻量索引器回扫全局上下文，只拉取相关远端片段）两条路径。
50%UnverifiedLinear attention methods (such as Performer and RWKV) compress complexity to O(N) via kernel function approximation, but their precise recall capability is typically significantly weaker than full attention.
50%UnverifiedMany linear attention or sparse attention methods lose the ability to precisely retrieve historical information while compressing computation.
50%UnverifiedGated Linear Attention (GLA) reduces computational complexity from O(N²) to O(N) through a linearized attention mechanism.
50%UnverifiedMinimax has trained a 456B-parameter MoE using a linear attention approach, but still requires occasional use of full attention.
50%UnverifiedLinear attention, sparse attention, and State Space Models (such as Mamba) can reduce computational complexity to linear levels, opening the door to million-token context windows.
50%UnverifiedDeltaNet's incremental update mechanism keeps GPU memory usage from growing with sequence length, maintaining a constant size instead, because it maintains a fixed-size hidden state rather than a full KV Cache.
50%UnverifiedHybrid architectures (linear attention + standard attention) are more efficient than pure full attention on long-context tasks, but linear attention may be slightly inferior to standard attention on certain reasoning tasks that require precise global dependencies
50%UnverifiedQwen3.8 Flash Next adopts a hybrid architecture whose core technology is Gated DeltaNet, combining linear attention with standard full attention
50%UnverifiedQwen3 27B uses linear attention in 48 of its 64 layers, with the remaining 16 layers paired with standard gated attention.
50%UnverifiedThe KV Cache footprint of linear attention is independent of sequence length, whereas the cache of traditional attention grows linearly with sequence length.
50%UnverifiedLinear attention is better suited to NPU chips due to the static graph characteristics of its fixed computation, whereas traditional attention uses dynamic graphs that require chips to frequently switch scheduling.
50%