Sliding Window Attention
A mechanism that restricts attention computation so that each token attends only to a fixed-size window of preceding tokens rather than the full sequence. This reduces attention complexity below O(n²) and is commonly used for memory optimization in long-context language models.
Core Facts
Timeline (last 90 days)
Sliding window attention lets each token attend only to neighboring tokens within a fixed-length window, thereby reducing attention computation and the KV cache from linear growth with sequence length to constant level.
Some of DeepSeek's model layers use sliding window attention, attending only to the most recent 128 tokens; SWA bounded replay replays only a short window at the end of the prompt, which is said to nearly eliminate total prefill work.
The architecture's sliding window is 128 tokens, and the KV cache of the window layers stops growing after 128 positions, no matter how long the context is.
For this architecture, the cached bytes per token is not a constant; the equivalent bytes/token of the window layers alone is about 688.0 at a context of 8,192, about 43.0 at 131,072, and about 5.4 at 1,048,576.
Muse Spark Open supports a 1-million-token context window, achieved through a hybrid attention architecture that alternates between sliding window attention and full attention across layers.
Sliding window attention is a local attention variant in the Transformer architecture, where each token only attends to tokens within a fixed-length window before and after it, reducing the attention computation complexity from O(n²) to O(n·w).
The implementation of unlimited context typically relies on retrieval-augmented generation (RAG) or variants of the sliding window attention mechanism.
Memory injection consumes context tokens, which is a trade-off of spending tokens to reduce tool-call conversation turns and the need to search through content.
Spark X 2.5 has 36 layers in total and uses a repeating '4-layers-per-group' structure, where each group contains three sliding window layers that attend only to the most recent 512 tokens and one global layer that reads the entire text
Industry solutions for improving long-context attention decay include sparse attention, sliding window attention, and RoPE positional encoding extrapolation, but currently no solution can completely eliminate this fundamental limitation.
1 more timeline events
All Facts (14)
Memory injection consumes context tokens, which is a trade-off of spending tokens to reduce tool-call conversation turns and the need to search through content.
65%UnverifiedGemma 4使用滑动窗口注意力（SWA）架构，该架构在处理数千行混淆JavaScript时导致模型遗忘早先读入的关键上下文
80%UnverifiedSliding Window Attention reduces the standard Transformer's O(n²) complexity to O(n·w), but the model cannot directly see information outside the window.
70%UnverifiedSliding window attention lets each token attend only to neighboring tokens within a fixed-length window, thereby reducing attention computation and the KV cache from linear growth with sequence length to constant level.
50%UnverifiedSome of DeepSeek's model layers use sliding window attention, attending only to the most recent 128 tokens; SWA bounded replay replays only a short window at the end of the prompt, which is said to nearly eliminate total prefill work.
50%UnverifiedThe architecture's sliding window is 128 tokens, and the KV cache of the window layers stops growing after 128 positions, no matter how long the context is.
50%UnverifiedFor this architecture, the cached bytes per token is not a constant; the equivalent bytes/token of the window layers alone is about 688.0 at a context of 8,192, about 43.0 at 131,072, and about 5.4 at 1,048,576.
50%UnverifiedMuse Spark Open supports a 1-million-token context window, achieved through a hybrid attention architecture that alternates between sliding window attention and full attention across layers.
50%UnverifiedThe implementation of unlimited context typically relies on retrieval-augmented generation (RAG) or variants of the sliding window attention mechanism.
50%UnverifiedSliding window attention is a local attention variant in the Transformer architecture, where each token only attends to tokens within a fixed-length window before and after it, reducing the attention computation complexity from O(n²) to O(n·w).
50%UnverifiedSpark X 2.5 has 36 layers in total and uses a repeating '4-layers-per-group' structure, where each group contains three sliding window layers that attend only to the most recent 512 tokens and one global layer that reads the entire text
50%UnverifiedIndustry solutions for improving long-context attention decay include sparse attention, sliding window attention, and RoPE positional encoding extrapolation, but currently no solution can completely eliminate this fundamental limitation.
50%UnverifiedThe KV Cache size of sliding window attention is constant, achieving O(1) memory usage rather than O(n).
50%Unverified滑动窗口注意力（如Mistral采用的方案）通过限制每个token只关注固定窗口内的历史，将复杂度从O(n²)降至O(n·w)
50%