Sliding Window Attention
A mechanism that restricts attention computation so that each token attends only to a fixed-size window of preceding tokens rather than the full sequence. This reduces attention complexity below O(n²) and is commonly used for memory optimization in long-context language models.
Timeline (last 90 days)
The implementation of unlimited context typically relies on retrieval-augmented generation (RAG) or variants of the sliding window attention mechanism.
Sliding window attention is a local attention variant in the Transformer architecture, where each token only attends to tokens within a fixed-length window before and after it, reducing the attention computation complexity from O(n²) to O(n·w).
Spark X 2.5 has 36 layers in total and uses a repeating '4-layers-per-group' structure, where each group contains three sliding window layers that attend only to the most recent 512 tokens and one global layer that reads the entire text
Industry solutions for improving long-context attention decay include sparse attention, sliding window attention, and RoPE positional encoding extrapolation, but currently no solution can completely eliminate this fundamental limitation.
The KV Cache size of sliding window attention is constant, achieving O(1) memory usage rather than O(n).
All Facts (7)
滑动窗口注意力将标准Transformer O(n²)的复杂度降至O(n·w)，但模型无法直接看到窗口之外的信息
95%UnverifiedGemma 4使用滑动窗口注意力（SWA）架构，该架构在处理数千行混淆JavaScript时导致模型遗忘早先读入的关键上下文
80%UnverifiedThe implementation of unlimited context typically relies on retrieval-augmented generation (RAG) or variants of the sliding window attention mechanism.
50%UnverifiedSliding window attention is a local attention variant in the Transformer architecture, where each token only attends to tokens within a fixed-length window before and after it, reducing the attention computation complexity from O(n²) to O(n·w).
50%UnverifiedSpark X 2.5 has 36 layers in total and uses a repeating '4-layers-per-group' structure, where each group contains three sliding window layers that attend only to the most recent 512 tokens and one global layer that reads the entire text
50%UnverifiedIndustry solutions for improving long-context attention decay include sparse attention, sliding window attention, and RoPE positional encoding extrapolation, but currently no solution can completely eliminate this fundamental limitation.
50%UnverifiedThe KV Cache size of sliding window attention is constant, achieving O(1) memory usage rather than O(n).
50%