Linear Attention
A class of attention mechanisms that replaces softmax attention with decomposable kernel functions, thereby reducing sequence complexity from O(n^2) to O(n).
Timeline (last 90 days)
Academic research has explored linear attention, Softmax-free attention, and kernel method approximations in multiple studies, attempting to replace the quadratic complexity of standard attention with linear complexity
Static or linearized attention alternatives have historically tended to make compromises in long-range dependency modeling and fine-grained semantic distinction
Hardware-level optimization techniques such as Flash Attention have significantly reduced actual memory overhead, but the fundamental quadratic bottleneck still exists
DeltaNet belongs to the linear attention mechanism family, designed to reduce the computational complexity of attention from O(n²) to O(n).
Linear attention generally performs well on long-context tasks but may fall slightly short of standard attention on certain tasks that require precise global dependencies.
All Facts (5)
Academic research has explored linear attention, Softmax-free attention, and kernel method approximations in multiple studies, attempting to replace the quadratic complexity of standard attention with linear complexity
50%UnverifiedStatic or linearized attention alternatives have historically tended to make compromises in long-range dependency modeling and fine-grained semantic distinction
50%UnverifiedHardware-level optimization techniques such as Flash Attention have significantly reduced actual memory overhead, but the fundamental quadratic bottleneck still exists
50%UnverifiedDeltaNet belongs to the linear attention mechanism family, designed to reduce the computational complexity of attention from O(n²) to O(n).
50%UnverifiedLinear attention generally performs well on long-context tasks but may fall slightly short of standard attention on certain tasks that require precise global dependencies.
50%