Sparse Attention
A technical approach for improving the Transformer attention mechanism by retaining only high-value attention connections rather than computing full pairwise token interactions, thereby reducing computational complexity from O(n²). Common implementations include sliding-window attention and hybrid global-plus-local attention, and the technique serves as a core method for enabling ultra-long context processing.
Core Facts
Timeline (last 90 days)
To maintain usable efficiency at the tens-of-millions context scale, it is usually necessary to introduce optimizations such as sparse attention, chunked retrieval, state space models or hybrid architectures.
Current mainstream sparse attention implementations can be divided into three categories: full reach, sliding windows, and learned selectors.
GLM-5.3-Flash 的注意力架构将计算拆为线性注意力（O(n)递归状态，处理近期token）和稀疏注意力（轻量索引器回扫全局上下文，只拉取相关远端片段）两条路径。
Sparse attention reduces computational complexity to near-linear by having each token attend only to a subset of the sequence.
Sparse attention may miss critical information for tasks requiring long-range dependencies, so in practice it is often combined with retrieval-augmented generation (RAG).
Adopting sparse attention can compress the KV cache to approximately 1/4, supporting approximately 4 times the context length of full attention under the same memory budget.
Many linear attention or sparse attention methods lose the ability to precisely retrieve historical information while compressing computation.
The paper reformulates sparse attention as a matrix approximation problem rather than a problem of selecting the largest values from a bag of entries.
Existing mainstream sparse attention methods filter interactions by retaining entries with larger values or high-quality regions in the attention matrix.
Sparse attention allows each query to interact with only k keys (where k is much smaller than n), reducing complexity to O(nk).
3 more timeline events
All Facts (13)
Informer reduces the self-attention computation complexity from O(n²) to O(n log n) through ProbSparse sparse attention.
70%UnverifiedTo maintain usable efficiency at the tens-of-millions context scale, it is usually necessary to introduce optimizations such as sparse attention, chunked retrieval, state space models or hybrid architectures.
50%UnverifiedCurrent mainstream sparse attention implementations can be divided into three categories: full reach, sliding windows, and learned selectors.
50%UnverifiedGLM-5.3-Flash 的注意力架构将计算拆为线性注意力（O(n)递归状态，处理近期token）和稀疏注意力（轻量索引器回扫全局上下文，只拉取相关远端片段）两条路径。
50%UnverifiedSparse attention reduces computational complexity to near-linear by having each token attend only to a subset of the sequence.
50%UnverifiedAdopting sparse attention can compress the KV cache to approximately 1/4, supporting approximately 4 times the context length of full attention under the same memory budget.
50%UnverifiedSparse attention may miss critical information for tasks requiring long-range dependencies, so in practice it is often combined with retrieval-augmented generation (RAG).
50%UnverifiedMany linear attention or sparse attention methods lose the ability to precisely retrieve historical information while compressing computation.
50%UnverifiedExisting mainstream sparse attention methods filter interactions by retaining entries with larger values or high-quality regions in the attention matrix.
50%UnverifiedSparse attention allows each query to interact with only k keys (where k is much smaller than n), reducing complexity to O(nk).
50%UnverifiedThe paper reformulates sparse attention as a matrix approximation problem rather than a problem of selecting the largest values from a bag of entries.
50%UnverifiedIndustry solutions for improving long-context attention decay include sparse attention, sliding window attention, and RoPE positional encoding extrapolation, but currently no solution can completely eliminate this fundamental limitation.
50%UnverifiedSpeculative decoding improves inference speed by using a small draft model to generate candidate tokens that are verified in parallel by the main model; sparse attention reduces computational complexity from O(n²) to near O(n).
50%