KV Cache as Runtime: The Third Path for AI Agent Interaction

Introduction: The Overlooked Third Path in AI Agent Development

When building large language model (LLM) agents, the industry typically focuses on two dimensions: model training and external scaffolding (prompt engineering, tool calling, multi-agent orchestration, etc.).

LLM agents are autonomous decision-making systems built on pre-trained language models like GPT, Claude, and Llama. Unlike traditional single-turn Q&A, agents can perceive their environment, plan tasks, use tools, and engage in continuous interaction. For example, AutoGPT can decompose complex tasks and autonomously invoke search engines, code executors, and other tools to achieve objectives. The core challenge for agents lies in maintaining contextual coherence across multi-turn interactions, making sound decisions, and collaborating efficiently with external systems.

However, the Yandex research team recently put forward an inspiring perspective — between these two dimensions lies a long-underestimated axis: the model's inference runtime.

Their core idea is this: by directly manipulating the internal state of the model during inference — the KV cache — you can achieve stronger interactivity and responsiveness. This idea extends the technical lineage of two earlier works from the same lab: Hogwild! Inference and AsyncReasoning.

The Essence of KV Cache: From Performance Optimization to Runtime Abstraction

How KV Cache Works

To understand this research, you first need to grasp the role of the KV cache in Transformer inference.

The Transformer is the core architecture of modern LLMs, proposed by Google in 2017. Its key innovation is the Self-Attention mechanism: when processing each token, the model computes relevance weights against all other tokens in the sequence. Specifically, each token generates three vectors — Query, Key, and Value. The attention score is computed via the dot product of the Query with all Keys, and these scores are used to produce a weighted sum of all Value vectors, yielding the contextual representation for that token. This process is repeated at every layer and every generation step, with a computational complexity of O(n²).

When an LLM generates text token by token, each new token requires attention computation with all preceding tokens. Autoregressive generation is the fundamental way LLMs produce text: the model predicts the next token based on the existing token sequence, appends the newly generated token to the end of the sequence, then predicts the following token based on the updated sequence, and so on until an end token is generated. This process is inherently sequential — each new token depends on information from all previous tokens. For example, to generate "artificial intelligence is changing the world": the model first predicts "artificial," then predicts "intelligence" based on "artificial," then predicts "is" based on "artificial intelligence," and so forth.

To avoid redundant computation, the model caches the Key and Value vectors corresponding to historical tokens — this is the KV cache. This sequential nature causes long-text generation to be slow, which is precisely the motivation for KV cache optimization.

Conventionally, the KV cache is viewed solely as a performance optimization — making autoregressive generation efficient. But the Yandex team's insight is that the KV cache actually carries the model's complete inference state. Since it is the physical carrier of that state, directly modifying it is tantamount to directly intervening in the model's "thinking process."

Paradigm Shift: Programmable Inference State

This shift in perspective is profoundly significant. If we view the KV cache as a programmable, editable runtime state — rather than merely a cache — we can dynamically alter the model's behavioral trajectory without retraining or relying on cumbersome external frameworks. This is exactly what the research team calls the "middle layer between model and scaffolding."

Limitations of Existing Approaches

The research team posed a pointed question: if modifying scaffolding is too abstract, and swapping or retraining models is too expensive — do we need a solution that sits between the two?

This question strikes at the heart of current agent development pain points:

Scaffolding level: Adjusting prompts, tool chains, or orchestration logic is flexible, but fundamentally applies influence "outside the model," unable to touch the model's internal reasoning dynamics, with limited improvements to interactivity and real-time responsiveness. External scaffolding refers to the augmentation layers built around the base model, including prompt engineering (guiding model behavior through carefully designed instructions), tool-calling frameworks (such as LangChain and LlamaIndex, enabling models to invoke APIs, databases, and other external resources), and multi-agent orchestration systems (solving complex problems through collaboration among multiple specialized agents). The advantage of these approaches is rapid iteration without modifying the model itself, but the downside is increased system complexity and limited control over the model's internal reasoning process.
Model level: Fine-tuning or retraining can fundamentally change capabilities, but at high cost, with long iteration cycles, making it difficult to handle scenarios requiring immediate adjustments.

The KV cache runtime fills precisely this gap: it provides low-cost, fine-grained, real-time intervention capability, allowing developers to dynamically inject, modify, or reorganize model state while inference is underway.

Technical Evolution: From Concurrent Inference to Asynchronous Reasoning

Hogwild! Inference: Concurrent KV Cache Operations

This work explored the possibility of having multiple inference processes share and concurrently operate on the same KV cache. Traditional autoregressive generation is strictly sequential, but the Hogwild! approach borrows the "lock-free concurrency" concept from parallel computing, allowing multiple inference streams to collaborate asynchronously, thereby improving overall interaction efficiency.

Hogwild! is a lock-free parallel stochastic gradient descent algorithm proposed in 2011. Its core idea is to allow lock-free concurrent updates to shared parameters during multi-threaded training — even if occasional conflicting overwrites occur, as long as the collision probability is low enough, the algorithm can still converge and run faster. This "wild growth" strategy sacrifices strict consistency in exchange for significant parallel speedup. The Yandex team brought this concept into the inference domain, allowing multiple inference processes to concurrently read from and write to the same KV cache, exploring higher interaction throughput under acceptable collision risk.

AsyncReasoning: Dynamically Interruptible Inference

AsyncReasoning further introduced the asynchronous concept into the reasoning process, making the model's "thinking" no longer a rigid single-threaded pipeline, but rather a dynamic process that can be interrupted, injected into, and reorganized by external events. This lays the foundation for building truly real-time responsive agents.

Together, these two works validated a key premise: the KV cache can be safely and effectively manipulated, and such manipulation can deliver interaction capabilities that traditional methods struggle to achieve.

Technical Validation: A Qwen3 Agent Playing DOOM in Real Time

The most striking demonstration in the blog is an agent based on the Qwen3 27B model, playing the DOOM game environment in real time using similar techniques.

The value of this demo goes far beyond the surface novelty of "AI playing games." DOOM (released in 1993) is a classic first-person shooter and a standard testing environment for AI research. Through open-source interfaces like ViZDoom, researchers can have AI agents receive visual input (screen pixels), game state (health, ammo, etc.), and output action commands (move, shoot, turn). The challenges DOOM poses for AI include: (1) real-time decision-making, where latency exceeding 100 milliseconds impacts gameplay; (2) spatial reasoning and long-term planning; and (3) handling rapidly changing dynamic scenes.

As a classic real-time interactive environment, DOOM is extremely sensitive to response latency — the player (or agent) must make decisions and react at the millisecond level. The fact that an agent driven by KV cache runtime manipulation can handle such a task demonstrates the viability of this approach in low-latency, high-interactivity scenarios. Compared to turn-based games like Go, DOOM is a far better test of an agent's real-time responsiveness — exactly the core advantage that KV cache runtime technology aims to showcase.

In other words, it demonstrates that runtime design itself can serve as an independent axis for enhancing agent capabilities.

Runtime Design: A New Dimension of Agent Capability

The core thesis put forward by the research team deserves serious consideration by the entire community: Is the design of model inference and runtime an underexplored axis of agent capability?

For a long time, the industry's progress narrative has revolved almost entirely around "bigger models" and "smarter scaffolding." But this series of works from Yandex suggests that beyond these two established dimensions, how we organize and manipulate the inference process may be an equally critical lever for capability improvement.

If this direction proves out, the implications would be far-reaching:

Cost efficiency: Unlocking new capabilities through runtime innovation alone, without the need to retrain expensive large models
Real-time performance: Opening new possibilities for applications requiring instant interaction (games, robotics, real-time conversation)
Composability: The runtime layer, as an independent abstraction, can be flexibly combined with different models and different scaffolding

Conclusion

Repositioning the KV cache from a pure performance optimization mechanism to an "agent runtime" represents an important conceptual leap. It reminds us that while pursuing parameter scale and ingenious frameworks, we should not overlook the tremendous plasticity inherent in the inference process itself.

Of course, this direction is still in its early exploration phase, and real-time KV cache manipulation faces numerous challenges in stability, generality, and production engineering that remain to be solved. But as the DOOM demo suggests, runtime design may well be the missing middle layer connecting "expensive models" and "abstract scaffolding." It deserves significantly more attention from researchers and engineers alike.