Unverified50% confidenceFactExact time
Linear attention replaces exp(q^T k) in softmax with a kernel function φ(q)^T φ(k), reducing overall complexity from O(n²d) to O(nd²).
1
Sources
50%
Confidence
Long-term
Relevance
10/12/2026
First Seen
Sources
Related Entities
Related Claims
VerifiedThe core formula of Self-Attention is Attention(Q,K,V) = softmax(QKᵀ/√dk)V, where dividing by √dk is intended to mitigate the vanishing gradient problem caused by excessively large dot product values.76% similarUnverifiedGated Linear Attention (GLA) reduces computational complexity from O(N²) to O(N) through a linearized attention mechanism.75% similarVerifiedStandard self-attention has a computational complexity of O(n²); SSA reduces complexity by having each token attend only to the most relevant subset, theoretically compressible to O(n·log n) or even O(n) level.74% similarUnverifiedLinear attention methods (such as Performer and RWKV) compress complexity to O(N) via kernel function approximation, but their precise recall capability is typically significantly weaker than full attention.74% similarVerifiedSliding window attention reduces the complexity of attention computation from O(n²) to O(n·w), linearizing GPU memory overhead for long texts.73% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/1002302API
curl https://kongchang.com/api/v1/knowledge/claims/1002302MCP
get_claim(id=1002302)