KVFetch: Filling the Missing Temporal Recall Channel in KV Cache Compression

The Hidden Flaw in KV Cache Compression

As the context windows of large language models expand to tens or even hundreds of thousands of tokens, KV cache compression has become an indispensable technique for efficient inference. The dominant approaches fall into three categories: score-based eviction, summary compensation, and offload-and-recall. Despite their differing designs, all three share a common underlying assumption — they decide which tokens to retain or recall based on relevance to the current query.

A newly published arXiv paper (arXiv:2610.08811) argues that this shared design philosophy harbors a structural flaw. Caches fundamentally support two access patterns: associative lookup by content, and sequential traversal by position. Existing compressors only implement the former, completely ignoring the latter.

The KV cache (Key-Value Cache) is the core data structure of the attention mechanism in Transformer architectures. During autoregressive inference, every time the model generates a new token, it must compute attention against all prior tokens in context. To avoid redundant computation, the model caches the Key and Value matrices at each layer for direct lookup by subsequent tokens. However, KV cache memory grows linearly with context length — for LLaMA-3-8B, processing 100K tokens can require tens of gigabytes, far exceeding the VRAM capacity of typical GPUs. This has led researchers to develop a range of compression strategies: score-based eviction methods (such as SnapKV and H2O) use attention scores to identify "unimportant" tokens and discard them; summary compensation methods capture the semantics of evicted content in an aggregated vector before discarding; and offload-and-recall methods migrate cold data to CPU memory or SSD, fetching it back to the GPU on demand. All three approaches share the same starting point: "content importance" — whether a token is likely to be frequently accessed given the current query.

What Is "Sequential Forgetting"

This gap has significant real-world impact. Tasks like retrieval-augmented generation (RAG), code completion, and structured data extraction all require the model to verbatim reproduce identifiers, field values, or code tokens from context. Under compression, content-based eviction mechanisms tend to retain the beginning of a sequence while discarding its continuation.

The result: verbatim copying breaks irreversibly midway through. The paper names this failure mode sequential forgetting.

The authors emphasize that this class of failures cannot be fixed by "better scoring algorithms," "larger budgets," "summary compensation," or "dynamic re-scoring." In other words, it is the dominant residual source of quality loss in existing compression schemes — a problem systematically overlooked by the entire research direction. This also echoes the paper's subtitle: "the other half of KV cache compression."

Understanding the root cause of sequential forgetting requires distinguishing between two fundamentally different ways the attention mechanism is used. During semantic comprehension, the model uses query vectors to retrieve semantically relevant fragments from the global context — a classic associative lookup where position is irrelevant and semantic similarity drives retrieval. During verbatim reproduction, the model is essentially performing a "relay" operation: it first locates the start of a sequence, then reads each subsequent token in positional order. These two patterns are structurally distinct. The scoring mechanisms used by existing compressors — whether attention averages, recency frequencies, or gradient signals — are designed for associative lookup. They naturally favor semantically salient tokens, while the middle fragment of a URL, the latter half of a hash string, or the tail of a line of code tends to score very low and is easily evicted. Once a token in the middle of a sequence is removed, any subsequent verbatim copying breaks at that point, with no recovery possible.

How KVFetch Works

To address this problem, the authors propose KVFetch — a training-free, drop-in framework that adds a temporal recall channel to any score-based compressor. Its core mechanism breaks down into three steps:

Hot/Cold Tiering

Rather than discarding eviction candidates outright, KVFetch demotes them to a quantized cold tier. This means the information does not vanish entirely — it is retained at a lower storage cost, leaving room for potential future recall.

In the context of KV cache compression, quantization typically refers to reducing floating-point precision from FP16 or BF16 to INT8 or even INT4, compressing the storage footprint of each KV vector to one-half or one-quarter of its original size. Compared to direct eviction, demotion via quantization is a "lossy retention" strategy: precision is reduced, but the position and rough semantics are preserved. KVFetch's design of combining a cold tier with quantization is well-targeted — in sequential access scenarios, the read pointer cares about whether a token at a given position exists, not about its high-precision representation. The precision loss from quantization therefore has a relatively limited impact on recall accuracy. This is the key to how the cold tier achieves a balance between storage efficiency and recall usability.

Monotone Read Pointer Detection

The framework uses a monotonically increasing read pointer to detect when the model is in an "active copy" state. When the model begins verbatim reproduction of context content, this pointer advances along positional order, identifying the occurrence of sequential access behavior.

Positional Successor Prefetching

Once a copy behavior is detected, KVFetch prefetches the positional successors of the accessed tokens into fixed-size hot-tier slots. Critically, this process adds no cost to attention computation, because the hot-tier slots are fixed in size and do not impose additional burden on the attention mechanism.

The elegance of this design lies in its decoupling of content-based recall from position-based recall, allowing each component to do what it does best — the existing scoring mechanism continues to handle content relevance, while KVFetch specifically fills in the temporal dimension of recall.

Experimental Results: Verbatim Copy Score Restored from 0.8 to 78.4

KVFetch's effectiveness is striking in benchmark evaluations. In iso-budget comparisons on the RULER-16K dataset:

• Verbatim copy scores recovered from 0.8 to 78.4 — nearly complete restoration;
• Average scores across 13 tasks improved by +8.4;
• Performance gains were concentrated on tasks requiring sequential access.

These numbers directly demonstrate that sequential forgetting is the dominant driver of compression quality loss — once this channel is added, copy capability can be restored from near-total collapse to functional levels.

Equally noteworthy is the framework's adaptive behavior. On LongBench, where no tasks require sequential access, the temporal recall channel remains dormant and incurs zero overhead. This shows that KVFetch is not a one-size-fits-all solution — it activates on demand based on task characteristics, avoiding unnecessary performance drag when not needed.

RULER (Realistic Unified Long-context Evaluation and Retrieval) is a benchmark designed specifically to evaluate long-context model capabilities, covering task types including single-hop and multi-hop retrieval, question answering, summarization, and verbatim copying, typically evaluated at context lengths ranging from 16K to 128K. "Iso-budget" comparison means the experiment compares with and without KVFetch under the same KV cache compression ratio (i.e., retaining the same number of hot-tier tokens), ruling out performance gains from increased budget and making the comparison more rigorous. LongBench is another widely used long-text understanding benchmark, with tasks primarily centered on QA, summarization, and code completion, with relatively few scenarios requiring precise verbatim output — making it a natural control group for validating KVFetch's on-demand activation behavior.

Implications for Long-Context Inference

From an engineering perspective, KVFetch's "training-free, drop-in" nature is its most practically valuable property. It requires neither retraining the model nor replacing existing compressors — it simply attaches as an enhancement layer on top of existing systems. For teams deploying long-context inference services, this means a long-overlooked pain point can be addressed at relatively low integration cost.

This work also offers an important research perspective: cache compression should not focus solely on "which content matters more" — it also needs to consider "how the model accesses that content." When application scenarios involve precise reproduction — whether of code, database fields, or retrieved document fragments — positional order itself is an information structure that must be preserved. As applications requiring verbatim output, such as RAG and code assistants, become increasingly widespread, optimizations targeting sequential access may well become standard components of the long-context inference stack.