Cache Engineering: Breaking Through the Long-Task Bottleneck of LLM Agents

As large language model (LLM) agents handle increasingly complex tasks, the trajectories of observations and actions they generate grow ever longer, and the I/O and storage overhead of key-value (KV) caches gradually becomes a system bottleneck. Traditional methods reduce the cache through context engineering—trimming or summarizing history—but this breaks the continuity of the trajectory prefix, causing the retained context to require re-prefill, which both increases latency and may sacrifice task accuracy. A recent paper proposes a new approach called "Cache Engineering": letting agents directly manipulate the active KV cache without modifying the trajectory or re-prefilling.

The Core Mechanism of Cache Engineering

Cache engineering centers on two operations: drop and repos. Drop allows the agent to actively exclude selected caches from future attention computations; repos adjusts the positional encoding of retained keys. These two operations act directly on the KV cache without touching the original trajectory, thereby avoiding the prefix invalidation problem.

The paper formalizes these operations and identifies a key serving challenge: historical dependency. The same token sequence may produce different cache states under different sequences of cache operations, which means prefix matching alone cannot correctly reuse caches. To address this, the authors designed the AKV serving engine, which internally manages stateful cache reuse, eviction, and restoration while exposing a stateless interface that is compatible with existing serving APIs.

KV Cache and Prefill: In large language model inference, the Transformer's self-attention mechanism needs to store key and value vectors for each generated token—this is the KV cache. When processing long trajectories, the cache grows linearly with sequence length. Prefill refers to the process in which the model performs a forward computation on the input sequence and builds the KV cache before generating the first token; if the cache is corrupted or the prefix is discontinuous, the model must re-execute prefill, which incurs significant latency and computational overhead. The goal of cache engineering is precisely to avoid this re-prefill.

Positional Encoding and RoPE: Modern LLMs commonly use rotary positional encoding (RoPE), which injects positional information by rotating key vectors. The repos operation adjusts the positional encoding of retained keys, essentially modifying their relative positions in attention computations, thereby reorganizing the cache without changing the original token sequence. This requires the serving engine to track and update the positional state of each key, increasing the complexity of cache management.

Performance Validation: Cost and Latency Both Drop

Experiments show that compared with traditional threshold-triggered summarization methods, adopting a rolling-drop cache strategy for tool responses improves the benchmark average score by 1.7 points and reduces average cost by 27.7%. On the AKV engine, output throughput improves by up to 1.2x, and P95 request latency is accelerated by 1.5x. More strikingly, recursive self-improvement of the cache strategy further reduces cost by 7.1% on top of expert design.

These data come from the experimental setup of a single paper, but they cover multiple benchmarks and are compared against mainstream summarization methods, making the conclusions reasonably convincing.

P95 Latency and Output Throughput: P95 request latency refers to the value within which 95% of requests complete, and is a key metric for measuring service response stability; output throughput refers to the number of tokens the system generates per unit time. Together, these two metrics reflect the efficiency of inference serving. In the experiments, a 1.5x acceleration in P95 latency means tail latency is significantly reduced, which is especially important for interactive agent applications.

Recursive Self-Improvement: This refers to the agent using its own runtime data or feedback to iteratively optimize its cache strategy—for example, by searching or learning to adjust the rule parameters of drop and repos. This mechanism allows the strategy to automatically evolve on top of expert design without human intervention, but note that its optimization effectiveness may be limited by the training data distribution and evaluation benchmarks.

Why Cache Engineering Is Worth Attention

One of the pain points in deploying agents today is the cost and latency of long contexts. Cache engineering provides a finer-grained level of control: rather than crudely truncating or rewriting history, it manages the KV cache like memory. This is especially critical for tasks that require multiple rounds of tool calls and long-trajectory planning.

In addition, the design philosophy of the AKV engine—stateless interface, stateful internals—lowers the barrier to integration with existing inference services. The open-source code has been released on GitHub, providing a reproducible foundation for future research.

In the future, automated optimization of cache strategies (such as recursive self-improvement) may become an important direction for agent self-evolution. However, the complexity of cache reuse brought by historical dependency still needs to be validated in more scenarios.

One-line Summary

Cache engineering lets agents directly manipulate the KV cache, cutting costs by 27.7% and improving throughput.

Paragraph Summary

The article introduces a new approach called "Cache Engineering," aimed at solving the KV cache overhead caused by long trajectories in large language model agents. Traditional context trimming or summarization breaks prefix continuity, leading to re-prefill. Cache engineering directly modifies the active KV cache through drop and repos operations, avoiding trajectory modification. The paper proposes the AKV serving engine, which internally manages stateful cache reuse and exposes a stateless interface. Experiments show that the rolling-drop strategy scores 1.7 points higher on average than threshold summarization and costs 27.7% less; the AKV engine improves throughput by 1.2x and accelerates P95 latency by 1.5x; recursive self-improvement further reduces cost by 7.1%. This direction provides a finer-grained means of cache control for long-context agents.