KVFetch
A training-free, plug-and-play KV cache compression framework that introduces a temporal recall channel to address the sequential forgetting problem present in existing compression schemes. Its core mechanisms include hot-cold tiered storage and a monotonic read pointer module.
Timeline (last 90 days)
KVFetch is a training-free, plug-and-play framework that can open a temporal recall channel for any score-based compressor.
The core mechanism of KVFetch operates in three steps: hot-cold tiering (demoting evicted tokens to a quantized cold tier), monotonic read pointer detection, and positional successor prefetching.
KVFetch prefetches the positional successors of accessed tokens into fixed-size hot-tier slots without increasing the attention computation cost.
In equal-budget controlled experiments on the RULER-16K dataset, KVFetch restored the verbatim copying score from 0.8 to 78.4.
In the RULER-16K experiments, KVFetch improved the average score across 13 tasks by +8.4, with gains concentrated on tasks requiring sequential access.
In LongBench testing, since no tasks require sequential access, KVFetch's temporal recall channel remains dormant and introduces no additional overhead.
The author believes that optimizations targeting sequential access are very likely to become standard components in long-context reasoning stacks.
All Facts (7)
KVFetch is a training-free, plug-and-play framework that can open a temporal recall channel for any score-based compressor.
50%UnverifiedThe core mechanism of KVFetch operates in three steps: hot-cold tiering (demoting evicted tokens to a quantized cold tier), monotonic read pointer detection, and positional successor prefetching.
50%UnverifiedKVFetch prefetches the positional successors of accessed tokens into fixed-size hot-tier slots without increasing the attention computation cost.
50%UnverifiedIn equal-budget controlled experiments on the RULER-16K dataset, KVFetch restored the verbatim copying score from 0.8 to 78.4.
50%UnverifiedIn the RULER-16K experiments, KVFetch improved the average score across 13 tasks by +8.4, with gains concentrated on tasks requiring sequential access.
50%UnverifiedIn LongBench testing, since no tasks require sequential access, KVFetch's temporal recall channel remains dormant and introduces no additional overhead.
50%UnverifiedThe author believes that optimizations targeting sequential access are very likely to become standard components in long-context reasoning stacks.
50%