Declarative Attention: Letting LLMs Control Their Own Attention, Boosting Long-Context Inference Efficiency by 52%

The Core Problem: The Computational Dilemma of Long-Context Attention

Current large language models face a fundamental contradiction when processing long contexts: the model's attention actually concentrates on only a few key tokens, yet it must scan the entire KV cache to locate this important information.

Background on Transformer Attention Mechanism Computational Complexity

Since the Transformer architecture was introduced in 2017, its core self-attention mechanism requires computing the relevance between every token and all other tokens when processing sequential data. This results in O(N²) computational complexity, where N is the sequence length. In practical deployment, models use KV caches (Key-Value Caches) to avoid redundant computation of historical token key-value pairs, but generating each new token still requires computing the dot product between the query vector and all historical key vectors in the cache — an O(N) complexity operation. As context lengths scale from thousands to millions of tokens, this linear scan becomes the primary bottleneck for inference latency and memory usage, limiting the practicality of long-context applications.

Consider a concrete example: when a user asks about a specific detail from earlier in a conversation containing 1 million tokens, the global attention layer must scan the entire context for every response token generated. This O(N)-level computational overhead severely constrains long-context inference efficiency and real-world usability.

Existing approaches attempt to pre-filter relevant tokens using lightweight proxy scoring, but these external scoring mechanisms still require a full scan at every step, failing to fundamentally solve the problem. This leads researchers to a fundamental question: Shouldn't the model itself know which parts of the context are relevant?

Declarative Attention Mechanism Design: Three Attention Declaration Modes

Declarative Attention (DA) takes an intrinsically-driven approach, allowing the model to actively declare which regions to attend to during chain-of-thought (CoT) reasoning.

Chain-of-Thought Reasoning and Intermediate Reasoning Steps

Chain-of-Thought (CoT) is a prompting technique that enables large language models to solve complex problems by generating intermediate reasoning steps. Unlike directly outputting an answer, CoT has the model unfold its reasoning process step by step — for example, "first identify the problem type → then locate relevant information → next analyze the evidence → finally draw a conclusion." This approach significantly improves accuracy on multi-step reasoning tasks. Declarative Attention leverages this mechanism by inserting attention declarations at different stages of the reasoning chain, allowing the model to explicitly indicate which context regions to attend to while generating reasoning steps, thereby concentrating computational resources on truly relevant areas.

This protocol divides the generation process into three modes:

<global> Mode: Full Context Access

Accesses the complete context, suitable for scenarios requiring global information — such as summarizing an entire document or answering questions that require cross-paragraph associations.

<focus> Mode: Precise Region Targeting

Focuses on specific regions, precisely locating relevant segments. When the model has already determined the approximate location of target information, this mode can dramatically narrow the attention scan range.

<local> Mode: Local Continuous Generation

Attends only to the most recent output, ideal for continuous generation tasks. During generation phases that don't require looking back at the context, this mode minimizes computational overhead to the greatest extent.

The inference engine processes these declarations much like parsing tool calls, thereby skipping most KV cache read operations. DA's key innovation is handing attention control to the model itself, leveraging the model's intrinsic understanding of context relevance to optimize computation paths rather than relying on external heuristic rules.

Zero-Shot Evaluation Results: The Efficiency-Accuracy Trade-off

The research team conducted zero-shot evaluations on off-the-shelf models across 15 long-context tasks, testing Gemma-4-31B and Qwen-3.6-27B. The results demonstrate remarkable efficiency gains:

Model	Attention Token Reduction	Accuracy Drop
Gemma-4-31B	52.0%	Only 1.27 percentage points
Qwen-3.6-27B	31.1%	2.75 percentage points

More notably, as model scale increases, the accuracy loss tends to shrink. This suggests that more capable models can more accurately determine which context regions require attention.

For practical application scenarios, a slight accuracy drop in exchange for nearly halving computational costs is a perfectly acceptable trade-off in most tasks such as document QA, long conversation summarization, and code analysis.

A New Direction for Sparse Attention Research

The Evolution of Sparse Attention Research

Sparse Attention is a core research direction for reducing Transformer computational complexity. Early approaches like Longformer used fixed sliding window + global token patterns, while BigBird employed a hybrid strategy of random attention + window attention. The limitation of these methods is that predefined sparsity patterns cannot adapt to the actual needs of different content. In recent years, learning-based dynamic sparsity methods have emerged, such as using lightweight networks to predict attention scores for token filtering, but these external scorers still need to scan the entire KV cache. Declarative Attention's innovation lies in completely delegating sparsity decisions to the model's own language understanding capabilities, using a symbolic protocol (similar to function calls) to let the model actively declare attention ranges during reasoning, achieving content-aware adaptive sparsification.

Declarative Attention opens an entirely new dimension for sparse attention research. Traditional sparse attention methods primarily rely on predefined patterns (such as sliding windows and fixed strides) or external heuristic rules, while DA enables models to dynamically determine attention allocation strategies based on actual content. This adaptive characteristic allows it to flexibly accommodate the specific needs of different tasks.

The paper notes that current zero-shot results are merely the starting point for this research direction. Future integration with training methods can further unlock DA's potential:

• Reinforcement learning optimization: Training models through RL to learn more precise attention declaration strategies
• Introducing DA during pre-training: Equipping models with autonomous attention control capabilities from the early stages of training
• Synergy with existing KV cache compression techniques: Stacking DA's benefits on top of existing cache optimizations

The KV Cache Compression Technology Ecosystem

KV cache optimization is another important pathway for accelerating large model inference. Existing techniques include: quantization methods that compress FP16 key-value pairs to INT8 or even INT4 to reduce memory usage; eviction strategies like H2O that dynamically discard unimportant historical tokens based on attention scores; and Grouped Query Attention (GQA) that shares a small number of key-value heads across multiple query heads to reduce cache size. These techniques primarily operate at the storage and transfer level, while Declarative Attention operates at the computational logic level, saving FLOPs by skipping attention computation over irrelevant regions. The optimization dimensions of both approaches are orthogonal and complementary, and can be used in combination: DA reduces the cache range that needs to be read, while quantization and eviction further compress the actually stored cache data, forming a multi-layered synergistic optimization.

From an architectural design perspective, DA's protocol-based approach carries broader implications: making control logic during inference explicit so it can be parsed and optimized. This paradigm is not only applicable to attention mechanisms but could also play a role in other aspects of model inference, providing a new conceptual framework for holistic optimization of LLM inference efficiency.