Prefix Caching
An LLM inference optimization technique that persistently caches the KV computation results of shared prefix content (such as system prompts) across multiple requests, enabling cross-request reuse to reduce redundant computation, lower latency, and improve throughput.
Timeline (last 90 days)
Prefix Caching enables KV reuse across requests; vLLM uses block hashing, while SGLang uses a radix tree.
PagedAttention supports multiple requests sharing the KV Cache of the same prefix (Prefix Caching).
vLLM uses Prefix Caching technology to reuse the KV Cache of requests with identical prefixes.
There is a bug in the current vLLM implementation: in prefix caching, the last cache block of approximately 16 tokens is directly discarded and recomputed, even when a request hits the cache.
When prefix caching and DFlash2 speculative decoding are both enabled simultaneously, total throughput drops by approximately 37%, which is even slower than not enabling speculative decoding at all.
Prefix caching can save 50%–80% of prefill-stage computation by reusing already-computed KV Cache, and vLLM implements fine-grained cache hit matching through a radix tree structure.
All Facts (4)
Prefix Caching enables KV reuse across requests; vLLM uses block hashing, while SGLang uses a radix tree.
50%UnverifiedPagedAttention supports multiple requests sharing the KV Cache of the same prefix (Prefix Caching).
50%UnverifiedvLLM uses Prefix Caching technology to reuse the KV Cache of requests with identical prefixes.
50%UnverifiedPrefix caching can save 50%–80% of prefill-stage computation by reusing already-computed KV Cache, and vLLM implements fine-grained cache hit matching through a radix tree structure.
50%