Prompt Cache
A mechanism in large model inference that caches and reuses repeated context such as system prompts, codebase backgrounds, and conversation history. It is a server-side extension of KV Cache and can significantly reduce costs and time-to-first-token latency.
Core Facts
The Prompt Caching mechanism can reuse the KV Cache of identical prefix contexts, typically reducing input token costs by 50%–90%.
Caching in large model services is generally divided into two levels: semantic caching and prompt prefix caching (KV Cache).
Timeline (last 90 days)
In this project, the number of times context was re-read (cache reads) was approximately 14 times the number of initial writes (inputs).
Mainstream large model API providers typically maintain prompt caches in units of 5 minutes to 1 hour, after which they expire.
Caching in large model services is generally divided into two levels: semantic caching and prompt prefix caching (KV Cache).
When multiple requests share the same prefix (such as a system prompt), efficient KV Cache reuse can significantly reduce computational overhead.
Prompt Cache refers to a provider caching already-processed identical prefix tokens, where the price of cache-hit tokens is typically only 10%–50% of the normal price.
For Agent applications with long system prompts or repeated use of the same documents, having the high-frequency repeated portions appear stably at the beginning of requests can improve cache hit rates and leverage lower pricing tiers.
When designing Agent prompts, content should be arranged along a stability spectrum: tool definitions and system rules at the front, knowledge base snapshots in the middle, and conversation history and user questions at the end, in order to maximize cache reuse.
The five main reasons for Prompt Cache misses include: the shared prefix is not long enough, computation identity is inconsistent, the cache has been evicted, the route misses the cache, and Decode accounts for the majority.
The standard for a cache hit is an exactly identical token prefix combined with an exactly identical computation identity, which includes the Tokenizer, Chat Template, model and weight version, and original multimodal input.
Prompt Cache operates on the computation of the Prefill phase, eliminating redundant prefix computation, but cannot skip the generation in the Decode phase.
2 more timeline events
All Facts (12)
The Prompt Caching mechanism can reuse the KV Cache of identical prefix contexts, typically reducing input token costs by 50%–90%.
80%VerifiedCaching in large model services is generally divided into two levels: semantic caching and prompt prefix caching (KV Cache).
75%UnverifiedIn this project, the number of times context was re-read (cache reads) was approximately 14 times the number of initial writes (inputs).
50%UnverifiedMainstream large model API providers typically maintain prompt caches in units of 5 minutes to 1 hour, after which they expire.
50%UnverifiedWhen multiple requests share the same prefix (such as a system prompt), efficient KV Cache reuse can significantly reduce computational overhead.
50%UnverifiedPrompt Cache refers to a provider caching already-processed identical prefix tokens, where the price of cache-hit tokens is typically only 10%–50% of the normal price.
50%UnverifiedFor Agent applications with long system prompts or repeated use of the same documents, having the high-frequency repeated portions appear stably at the beginning of requests can improve cache hit rates and leverage lower pricing tiers.
50%UnverifiedThe standard for a cache hit is an exactly identical token prefix combined with an exactly identical computation identity, which includes the Tokenizer, Chat Template, model and weight version, and original multimodal input.
50%UnverifiedThe five main reasons for Prompt Cache misses include: the shared prefix is not long enough, computation identity is inconsistent, the cache has been evicted, the route misses the cache, and Decode accounts for the majority.
50%UnverifiedWhen designing Agent prompts, content should be arranged along a stability spectrum: tool definitions and system rules at the front, knowledge base snapshots in the middle, and conversation history and user questions at the end, in order to maximize cache reuse.
50%UnverifiedPrompt Cache operates on the computation of the Prefill phase, eliminating redundant prefix computation, but cannot skip the generation in the Decode phase.
50%UnverifiedClaude's cache read pricing is approximately 2.5% of the standard input price.
50%