Prompt Caching
A pricing optimization mechanism introduced by Anthropic that applies token-level cost discounts to reusable fixed-prefix portions of requests, such as system prompts, passing on savings from KV cache reuse to the billing structure.
Core Facts
The Claude API supports a Prompt Caching mechanism, where content is billed at full price on the first request, and subsequent tokens that hit the cache cost approximately 10% of the original price
The Prompt Caching mechanism can reuse the KV Cache of identical prefix contexts, typically reducing input token costs by 50%–90%.
The core principle of prompt caching is cross-request reuse of the KV Cache, where already-computed KV states can be cached and reused when multiple requests share the same prefix.
Timeline (last 90 days)
Long context significantly raises time to first token, because prefill computation grows with input length; hitting the cache can skip most of the prefill and directly improve first-token latency.
Whether prompt caching hits depends on byte-for-byte consistency of the request prefix; if even one character at the beginning differs, the entire cache is invalidated.
The cache is not permanently valid; providers usually have a TTL of a few minutes to an hour, and the hit rate fluctuates with user behavior.
After switching to the new architecture, Legion's average cached volume dropped from 6.48M to 3.48M (-46%), and average uncached volume dropped from 1.71M to 1.03M (-40%).
Prompt Caching is a cost-optimization mechanism introduced by mainstream LLM providers (such as Anthropic and OpenAI): when the prefix portion of multiple requests is completely identical, the provider caches that portion of the computation, and tokens that hit the cache are billed at a price far below normal input.
Mainstream Prompt Caching solutions use character-level exact matching; an extra space, a changed punctuation mark, or a reordered paragraph all cause a cache miss.
Prompt Caching can greatly reduce first-token latency, and in long-prompt scenarios first-token speed can typically improve by 70% to 90%.
In bidirectional attention models (such as BERT encoders), each token's representation is affected by the entire text, so prefix KV cannot be reused independently; therefore Prompt Caching mainly appears on generative large models rather than encoder models.
Prompt Caching is an optimization across requests, users, and conversations: as long as the prompt prefixes of different requests are completely identical, the corresponding KV results can be shared globally.
The implementation logic of Prompt Caching is: hash-check the input prompt, match it against cached KV prefixes, and if there is a hit, skip the Transformer computation for that part and perform inference computation only on the suffix part.
40 more timeline events
All Facts (20)
The Claude API supports a Prompt Caching mechanism, where content is billed at full price on the first request, and subsequent tokens that hit the cache cost approximately 10% of the original price
80%VerifiedThe Prompt Caching mechanism can reuse the KV Cache of identical prefix contexts, typically reducing input token costs by 50%–90%.
80%VerifiedThe core principle of prompt caching is cross-request reuse of the KV Cache, where already-computed KV states can be cached and reused when multiple requests share the same prefix.
75%VerifiedIn Anthropic's prompt caching mechanism, cache-hit tokens are charged at only 10% of the standard price.
70%VerifiedSome service providers have introduced cache-hit pricing, charging at a lower discounted rate when repeated long system prompts hit the cache.
65%VerifiedIn cache hit scenarios, hit tokens are typically charged at 1/10 of the original price or even lower
65%VerifiedThe disk caching mechanism persists already-computed KV Cache to storage media; when a subsequent request's prefix hits the cache, it can be reused directly, skipping redundant forward propagation.
65%VerifiedThe general principle for achieving cache hits is to keep the prefix of the request context as stable as possible, placing fixed content such as system prompts and tool definitions at the very front, with per-turn user input appended at the end.
65%VerifiedBoth Anthropic and OpenAI have introduced discount policies related to prompt caching.
65%VerifiedPrompt Caching reduces latency and token costs by reusing the KV Cache of identical prefixes to skip redundant computation.
65%UnverifiedAnthropic's Prompt Caching can reduce repeated input costs by 90%
80%UnverifiedPrompt caching technology stores KV matrices in high-speed memory, and when the next request contains the same prefix content, it reads directly from the cache, skipping recomputation.
70%UnverifiedPrefix matching requires that only a contiguous sequence starting from the first token that is completely identical can hit the cache.
70%UnverifiedClaude's prompt caching is an optimization mechanism provided by Anthropic at the API level, where the price of cached read tokens is approximately one-tenth that of regular input tokens.
60%UnverifiedPrompt caching uses a strict prefix-matching strategy; once a mismatch occurs at a given position, all cached content after that position is invalidated.
60%UnverifiedWhether prompt caching hits depends on byte-for-byte consistency of the request prefix; if even one character at the beginning differs, the entire cache is invalidated.
50%UnverifiedLong context significantly raises time to first token, because prefill computation grows with input length; hitting the cache can skip most of the prefill and directly improve first-token latency.
50%UnverifiedThe cache is not permanently valid; providers usually have a TTL of a few minutes to an hour, and the hit rate fluctuates with user behavior.
50%UnverifiedAfter switching to the new architecture, Legion's average cached volume dropped from 6.48M to 3.48M (-46%), and average uncached volume dropped from 1.71M to 1.03M (-40%).
50%UnverifiedPrompt Caching is a cost-optimization mechanism introduced by mainstream LLM providers (such as Anthropic and OpenAI): when the prefix portion of multiple requests is completely identical, the provider caches that portion of the computation, and tokens that hit the cache are billed at a price far below normal input.
50%