Where Should KV Cache Live? A Deep Dive into GPU/CPU/SSD Tiered Placement Strategies

Long-session LLM inference is facing an increasingly thorny resource challenge: GPU High Bandwidth Memory (HBM) is scarce and expensive, while growing conversation turns, agent loops, and document QA workloads continuously accumulate context state — and KV cache is devouring precious GPU memory. A newly published arXiv paper (arXiv:2609.16215v1) systematically investigates a core question the industry has long overlooked: which storage tier should KV cache blocks actually live on?

Tiered Storage: A Practical Path to Extending GPU Memory

A number of systems have already attempted to extend GPU memory capacity using CPU DRAM and SSDs. Notable examples include Mooncake, LMCache, FlexGen, InfiniGen, and AttentionStore. The shared idea behind these systems is to build a multi-tier storage hierarchy spanning GPU HBM, CPU DRAM, and SSD — offloading KV cache that doesn't fit in GPU memory to cheaper, higher-capacity media.

But the real challenge isn't whether you can extend storage — it's the placement policy: which cache blocks should stay on the GPU, which should be moved to CPU or SSD? When should migration happen, and when should eviction occur? Does prefetching actually help? These decisions directly affect migration overhead and response latency.

KV cache (Key-Value Cache) is a core data structure in LLM inference. In Transformer architectures, computing attention requires every token to access the Key and Value vectors of all previous tokens in the sequence. Recomputing these vectors from scratch for every new token would result in quadratic scaling with sequence length. KV cache solves this by storing previously computed K/V vectors for direct reuse, reducing complexity to linear — but at the cost of memory usage that scales linearly with context length. For LLaMA-3 70B at FP16 precision, a single 32K-token context can produce tens of gigabytes of KV cache, far exceeding the capacity of a single GPU. This is the fundamental driver behind tiered storage solutions: GPU HBM is typically capped at around 80GB, while the cumulative KV cache generated by long sessions, multi-turn agents, and large document QA workloads can be several times — or even tens of times — larger.

Validating Strategies with a Discrete Event Simulator

The researchers built a discrete event simulator covering all three storage tiers — GPU HBM, CPU DRAM, and SSD — and calibrated it with a Random Forest execution time predictor to ensure simulation results closely match real-world runtime behavior.

Within this framework, they compared four placement strategies: Recency (recent access), Reuse Frequency, Predicted Reuse, and an EWMA (Exponentially Weighted Moving Average) predictor with lookahead prefetching. The evaluation covered three representative workload types — chat conversations, agent loops, and document QA — to capture the diverse access patterns seen in real deployments.

Discrete Event Simulation (DES) is a classic methodology in systems performance research. It abstracts system behavior as a sequence of discrete, time-ordered events (e.g., "KV block migration starts," "decode step completes") and advances an event queue to simulate system behavior under realistic workloads — without requiring actual hardware deployment. Compared to real-machine experiments, DES offers key advantages: it enables rapid exploration of large strategy-parameter combinations (this study covers a grid of 4 strategies × multiple cache capacity ratios × 3 workload types); and it allows the introduction of idealized baselines like an "Oracle prefetcher" — which couldn't exist in a real system — for upper-bound analysis. Its limitation lies in model accuracy depending on calibration quality, which is precisely why the researchers incorporated a Random Forest execution time predictor: to ensure simulated latency and migration overhead numbers align with measurements from real GPU inference systems.

The Gains from Tiering Come from Capacity, Not Strategy

The experimental results are striking: with tiered storage, the number of concurrent sessions supported by a single GPU increased by 73.02×, and cost per session dropped by 62.04×. These numbers make a compelling case for tiered storage in long-session inference.

But the study delivers a key counterintuitive finding: these gains come primarily from stacking capacity (a three-tier ratio of 1 + 8 + 64), not from sophisticated placement policies. In their experimental setup, the decode phase at batch size 1 is compute-bound, so placement strategy has almost no effect on throughput. What placement policy actually changes is PCIe migration traffic and Time to First Token (TTFT) — not the overall performance ceiling.

Different Workloads Favor Different Strategies

The study provides detailed strategy comparisons. For chat workloads, the Recency strategy generates 2.30× less migration traffic than the Reuse Frequency strategy and outperforms it overall. For agent and document QA workloads, Reuse Frequency is the better choice.

Also worth scrutinizing is the paper's treatment of existing "Predicted Reuse" strategies. The researchers found that the current Predicted Reuse strategy is byte-identical to Recency — meaning its recommendation for agent workloads is effectively just Recency. A true EWMA-based predictor does change behavior, but on the workloads where prediction was expected to help most, it still underperforms compared to simple Reuse Frequency.

Prefetching: More Bandwidth Cost Than It's Worth

Prefetching has long been viewed as a tool for hiding migration latency, but this research provides evidence against it. Across the full grid of strategies and cache size combinations, even an Oracle prefetcher with perfect knowledge of future requests never outperforms the no-prefetch baseline on migration traffic. In other words, the additional bandwidth consumed by prefetching is not justified by a corresponding benefit.

Prefetching is typically a powerful technique for reducing read latency in storage system design: data is proactively moved from a slower medium to a faster one, so it's already nearby when needed. In the KV cache context, the ideal scenario for prefetching would be: before the next request in a session arrives, anticipate the required KV blocks and preload them from SSD or DRAM into GPU HBM — eliminating the migration wait time's impact on TTFT. However, the paper's negative finding reveals a key bandwidth economics problem: prefetching inevitably causes "wasted transfers" — data loaded preemptively but never used occupies precious PCIe or memory bandwidth that could otherwise serve genuinely necessary migrations. Given that the decode phase is already compute-bound and bandwidth is not a bottleneck to relieve, the marginal benefit of prefetching is extremely limited. The extra traffic instead intensifies bandwidth contention, which is why even a perfectly informed Oracle prefetcher cannot achieve a net gain.

Implications for System Designers

The value of this research lies in demystifying prevailing assumptions. It clearly shows that workload-specific placement policies can reduce data movement — but "Predicted Reuse" and "prefetching," two approaches that have received considerable attention in the field, are not supported by experimental evidence as currently implemented.

For teams building long-session inference infrastructure, this suggests a few practical takeaways: prioritize expanding tiered capacity for maximum gains; use Recency for chat workloads to reduce migration traffic, and Reuse Frequency for agent and document QA; and approach complex mechanisms like predicted reuse and prefetching with skepticism — validate before investing. At a time when GPU memory costs remain high, focusing engineering effort on what actually works may matter more than chasing sophisticated prediction algorithms.