Prefix Caching
An LLM inference optimization technique that caches and reuses the KV Cache from requests sharing identical prefixes, thereby reducing redundant computation and improving processing efficiency for multi-turn conversations and batches of similar requests.
Timeline (last 90 days)
Estimating loop cost at flat input prices, under near-ideal caching conditions it was overestimated by about 7 times on average.
Among the 600 trajectories in this study, 98.6% of input tokens were provided by the prefix cache.
Mechanisms such as prefix caching share and deduplicate the cache by common prefixes to improve the reuse rate when multiple requests share a system prompt.
Prefix caching skips redundant computation by reusing previously computed KV cache for identical prefixes from prior requests, but any formatting change, order perturbation, or insertion of content in the middle will invalidate the cache.
When a prefix cache hit occurs, inference latency can drop from the second level to the millisecond level, and the cache-hit price offered by various vendors is typically one-tenth or even lower than the cache-miss price.
Prefix caching is extremely sensitive to input stability; as long as any single token in the prefix is modified or inserted, the cache is invalidated and all subsequent content must be recomputed.
The core update in v0.29.0 is a fix for the default configuration issue when hybrid models apply dense prefix caching.
Prefix caching improves inference efficiency by caching common prompt prefixes to reduce redundant computation.
Prefix caching can significantly improve inference efficiency when processing batch requests with the same prefix.
Prefix caching leverages the fact that the system prompt and conversation history remain unchanged across turns in long conversations, saving the KV cache of the fixed prefix for reuse in new turns to avoid redundant computation.
6 more timeline events
All Facts (14)
Prefix caching improves inference efficiency by caching common prompt prefixes to reduce redundant computation.
70%UnverifiedMainstream inference frameworks such as vLLM already natively support automatic prefix caching.
60%UnverifiedEstimating loop cost at flat input prices, under near-ideal caching conditions it was overestimated by about 7 times on average.
50%UnverifiedAmong the 600 trajectories in this study, 98.6% of input tokens were provided by the prefix cache.
50%UnverifiedMechanisms such as prefix caching share and deduplicate the cache by common prefixes to improve the reuse rate when multiple requests share a system prompt.
50%UnverifiedPrefix caching skips redundant computation by reusing previously computed KV cache for identical prefixes from prior requests, but any formatting change, order perturbation, or insertion of content in the middle will invalidate the cache.
50%UnverifiedWhen a prefix cache hit occurs, inference latency can drop from the second level to the millisecond level, and the cache-hit price offered by various vendors is typically one-tenth or even lower than the cache-miss price.
50%UnverifiedPrefix caching is extremely sensitive to input stability; as long as any single token in the prefix is modified or inserted, the cache is invalidated and all subsequent content must be recomputed.
50%UnverifiedThe core update in v0.29.0 is a fix for the default configuration issue when hybrid models apply dense prefix caching.
50%UnverifiedPrefix caching can significantly improve inference efficiency when processing batch requests with the same prefix.
50%UnverifiedPrefix caching leverages the fact that the system prompt and conversation history remain unchanged across turns in long conversations, saving the KV cache of the fixed prefix for reuse in new turns to avoid redundant computation.
50%UnverifiedvLLM's prefix caching implementation is based on a radix tree data structure for fine-grained cache hit matching.
50%UnverifiedWhen the cache hit boundary is not aligned with the block boundary, if the hit endpoint falls in the middle of a block, it triggers recomputation of the entire block, even though most tokens could have been reused.
50%UnverifiedPrefix caching can save 50%–80% of prefill-stage computation in multi-turn dialogue or batch processing scenarios.
50%