Persistent KV Cache Connector: An LLM Inference Optimization Approach for Cross-Request and Cross-Restart Reuse

A KV Cache Persistence Solution for LLM Inference

In large language model (LLM) inference, the Key-Value Cache (KV cache) is a critical factor in determining throughput and latency. A developer recently shared on Reddit a KV connector they built, with the core capability of persisting KV caches to disk so they can be reused across different requests and even after service restarts — and they're looking for community feedback.

While the project is currently just an early-stage prototype seeking input, the problem it tackles is familiar to any team that has deployed an inference service: recomputing KV caches for every request is expensive, and a service restart means losing all cached data.

Why KV Cache Persistence Is Worth Caring About

The Computational Cost of KV Caches

During autoregressive generation, Transformer architectures must maintain Key and Value tensors for every previously processed token. As context length grows, the GPU memory consumed by the KV cache increases dramatically, and the compute required during the prefill phase is substantial. For applications with long system prompts or shared prefixes, repeatedly computing the same prefix cache is an obvious waste.

Understanding the scale of this overhead helps evaluate the value of persistence. Take LLaMA-3 70B as an example: the KV cache size per token is roughly 2 × num_layers × head_dim × bytes_per_element. With 80 layers, head_dim=128, and BF16 precision, each token occupies about 80KB — meaning a 4,096-token long context requires approximately 320MB of GPU memory. The computational complexity of the prefill phase scales quadratically with sequence length (O(n²)), so for a high-concurrency service where every request shares the same 2,000-token system prompt, running prefill from scratch each time means repeatedly paying for the same compute. This is exactly why prefix caching is already widely supported in frameworks like vLLM and SGLang — but these implementations typically only live within the process lifetime and are lost on restart.

The Value of Cross-Request Reuse

The first pain point this connector addresses is cross-request reuse. When multiple requests share the same context prefix — such as identical few-shot examples or the same document context — caching that portion of the KV data and loading it directly can significantly reduce redundant prefill computation, lowering time-to-first-token (TTFT).

The Significance of Cross-Restart Persistence

The second pain point is cache loss after service restarts. Conventional KV caches live in GPU memory or host RAM and are wiped when the process exits. Persisting the cache to disk means that after a rolling upgrade, crash recovery, or elastic scaling event, hot caches can be restored — reducing the performance jitter caused by cold starts.

The Real Engineering Challenges of Disk Persistence

Writing KV caches to disk sounds straightforward, but there are quite a few engineering trade-offs to consider:

• Storage volume: KV caches are large — a single request with a long context can reach several GB, making disk I/O and capacity management genuine concerns.
• I/O bandwidth: If loading a cache from disk is slower than recomputing it, reuse loses its value. The random read performance of NVMe SSDs, and whether compression or quantized storage is needed, directly affects the benefit.
• Cache invalidation and consistency: Any change to model weights, quantization precision, or attention implementation renders old caches invalid. A reliable key-matching and versioning mechanism is essential.
• Hit rate: The real-world value of a persistence scheme is highly dependent on hit rate. If request prefixes are highly diverse, low cache reuse rates may mean the overhead of persisting to disk outweighs the gains.

What Feedback the Community Can Offer

The author explicitly states this is a project looking for feedback. For an early-stage tool like this, the most valuable community input typically falls into a few areas:

1. Benchmarking: In real-world workloads, how much latency and compute does loading from disk actually save compared to rerunning prefill? Quantitative comparisons of TTFT and throughput are needed.
2. Integration ecosystem: Is it compatible with mainstream inference frameworks (such as vLLM's KV connector interface)? Can it be seamlessly plugged into existing serving stacks?
3. Tiered storage strategy: Compared to going straight to disk, a multi-level cache combining GPU memory, host RAM, and disk may be more practical — keeping hot data in upper tiers and cold data on disk.
4. Compression and quantization: Does the solution apply compression or reduced-precision storage to persisted caches to ease storage and bandwidth pressure?

Summary

This KV connector represents an increasingly important direction in LLM inference optimization — treating the KV cache as a reusable, persistent asset rather than a single-use throwaway. From shared prefix reuse to post-restart recovery, tools like this have the potential to further reduce inference costs in production environments.

However, given the limited information in the original post — no concrete performance data or implementation details — its real-world effectiveness still needs to be validated through more thorough benchmarking. For engineering teams focused on inference infrastructure, this is a community project worth watching.

Additional Context

Starting with version 0.6.x, vLLM introduced a standardized KV Connector plugin interface (KVConnectorBase), allowing third-party implementations to offload KV caches to external storage — whether a remote memory pool (e.g., Redis, Memcached), object storage, or local disk. This interface defines standard methods such as send_kv_caches_and_hidden_states and recv_kv_caches_and_hidden_states, enabling connectors to inject cache routing logic without modifying vLLM's core inference logic. If this project is compatible with that interface, it means it could be enabled as a plugin in existing vLLM deployments without forking or deeply customizing the inference engine. This is a key reference point for evaluating its integration value.

Tiered caching already has mature precedents in industry: CPU host memory bandwidth is roughly 1/5 to 1/10 that of GPU memory, while NVMe SSD sequential read speeds (around 7GB/s) are another one to two orders of magnitude lower than host memory. A sensible cache hierarchy therefore looks like this: hot prefixes stay in GPU HBM, moderately hot data offloads to host DRAM (similar to CPU KV cache offloading), and cold data is written to disk. The break-even point for whether loading from disk is faster than recomputing depends on the ratio between the model's prefill compute throughput (TFLOPS) and available storage I/O bandwidth — for large, compute-intensive models, the window where disk loading wins is relatively wider; for smaller models, that may not hold.