Tsinghua Open-Sources AKV: A Cache Engineering Server for LLM Agents on SGLang

From "Deleting History" to "Managing Cache": A New Approach for LLM Agents

Long-context agents face an awkward reality: the longer the conversation history, the more bloated the KV cache becomes, and inference cost and latency rise in tandem. The common industry practice is to have the model summarize history or simply truncate it, but both approaches lose critical information. A research team at Tsinghua University offers another path—treating the KV cache itself as an engineerable resource to be managed—and has implemented an open-source serving system, AKV, on top of SGLang.

According to MarkTechPost, AKV allows agents to "discard" and "relocate" KV cache during inference rather than relying on summarization to compress history. Official figures show that on 4 agent benchmarks, the approach reduces average cost by 27.7%.

What Problem Does AKV Solve?

Cache Bloat Is a Hidden Cost for Agents

In multi-turn agent tasks, each inference step must carry the context accumulated so far. Traditional approaches pass the entire history as an indivisible KV block with the request, causing memory usage and prefill computation to grow continuously. When history exceeds the window limit, the system can only discard early content or have the model generate summaries—the former may lose task-critical clues, while the latter introduces extra generation overhead and information loss.

AKV's approach is to treat the cache as an editable data structure: researchers can mark which cache segments should be retained, which can be discarded, and which need to be moved to new positions. This "cache engineering" allows agents to significantly compress irrelevant history while preserving task-relevant context.

KV cache is a key optimization in the Transformer architecture during autoregressive generation: each time a new token is generated, the model needs the Key and Value vectors of all previous tokens. To avoid redundant computation, the inference engine caches these intermediate results. For multi-turn agent tasks, the KV cache corresponding to the entire conversation history grows linearly with the number of turns, not only consuming substantial GPU memory but also forcing the prefill phase of each new request to process longer sequences, directly driving up latency and compute cost. Cache bloat is therefore not merely a storage problem but a core bottleneck affecting inference economics.

Comparison with the "Summarize History" Approach

Summarizing history relies on the model's generation capability, essentially compressing long context into short text at the cost of information loss and an additional model forward pass for each compression. AKV operates directly on the KV cache, eliminating the summarization step and avoiding the risk of task failure caused by a summarization model's misunderstanding. From an engineering perspective, this is more like database index management than natural language processing.

SGLang is a high-performance inference serving framework for large language models, and one of its core innovations is RadixAttention: it organizes KV cache using a radix tree, enabling automatic sharing of cache with identical prefixes across multiple requests to avoid redundant computation. This mechanism is naturally suited to multi-turn dialogue scenarios, since subsequent turns typically reuse prior history. AKV extends SGLang with cache editing capabilities, allowing agents not only to reuse cache but also to actively discard or relocate cache segments—upgrading cache management from passive reuse to active engineering control.

Technical Implementation and Open-Source Form

AKV is built on SGLang. SGLang itself is a high-performance serving framework for large model inference, featuring cache reuse mechanisms such as RadixAttention. AKV adds agent-oriented cache editing interfaces on top of this foundation, enabling developers to programmatically control the lifecycle of KV cache.

For agent developers, this means cache policies can be declared directly at the task orchestration layer without modifying the model itself or relying on external summarization modules. The open-source release also lowers the barrier to reproduction, allowing the research community to compare the effects of different cache strategies on the same serving system.

What Does a 27.7% Cost Reduction Mean?

In multi-agent or long-horizon tasks, inference cost often grows approximately linearly with the number of turns. An average cost reduction of 27.7% is no small number—it means the same budget can support more interaction rounds, or that API and compute costs can be significantly reduced for the same task volume.

Objectively, this figure is an average across 4 agent benchmarks; benefits may vary noticeably by task type. The extent to which cache can be discarded depends on how strongly a task relies on historical information: information-retrieval tasks may benefit more, while tasks requiring strict tracking of early instructions may not.

Implications for Agent Infrastructure

AKV represents a trend: agent optimization is shifting from prompt engineering down to the inference system layer. Over the past year, research on KV cache compression, paged management, and prefix reuse has emerged密集ly, indicating that long-context cost has become one of the core bottlenecks for agent deployment.

The Tsinghua team's choice to open-source on SGLang rather than commercialize it closed-source also helps foster a tooling ecosystem around cache engineering. If more frameworks adopt similar cache editing primitives in the future, agent developers may be able to configure context policies much like configuring database indexes—turning "what to remember and what to forget" into tunable parameters rather than a black art.

Currently, public information on AKV remains focused on the methodology level; specific interface design, scheduling strategies, and adaptation results across different models still need to be validated by the community through practical use.

One-line Summary

Tsinghua open-sources AKV, reducing agent inference cost by 27.7% by editing KV cache rather than summarizing history.

Paragraph Summary

The Tsinghua University research team has open-sourced AKV, an LLM agent serving system built on SGLang, aimed at solving the cost and latency problems caused by KV cache bloat in long-context agents. Traditional methods compress context by summarizing history or truncation, but they lose critical information or introduce extra overhead. AKV treats KV cache as an editable data structure, allowing agents to discard and relocate cache segments during inference, thereby compressing irrelevant history while preserving task-relevant context. On 4 agent benchmarks, AKV reduces average cost by 27.7%. This work represents the trend of agent optimization shifting from prompt engineering down to the inference system layer, providing a new engineering approach to the long-context cost problem.