Prompt Caching

Whether prompt caching hits depends on byte-for-byte consistency of the request prefix; if even one character at the beginning differs, the entire cache is invalidated.

Prompt cache validity typically ranges from only a few minutes to several hours, and discount rates and cache windows may be adjusted at any time.

The cache read mechanism retains the KV cache for recurring long prompt prefixes, charging at an extremely low rate upon a cache hit; the leaked cache read price for Opus 5.5 is approximately 1/20 of the input price.

In Anthropic's API billing system, prompt caching allows a stable prefix to be marked as cached content, and subsequent identical prefixes directly reuse the KV without recomputing attention, with these tokens recorded as "cache reads".

Prompt caching relies on the Transformer's KV cache mechanism, skipping redundant computation by reusing already-computed KV vectors for identical prefixes, saving GPU compute and reducing time-to-first-token latency.

The 12-hour cache strategy is especially suited to scenarios such as customer service bots, content creation assistants, coding assistants and complex multi-turn conversations.

The Reddit user discussion reflects the widespread demand among users for more flexible control over cache duration to suit different workflows.

Different AI providers differ in cache granularity, hit conditions and billing methods; for example, some price cache-write tokens and cache-read tokens separately, and some set a minimum cacheable token length.

The original Reddit post describes that users now face three choices: stick with the default 30-minute cache, switch to a 12-hour cache, or use a mix of the two.

The default 30-minute prompt cache suits fast, continuous conversations.

The cache pricing strategies of mainstream AI providers (such as OpenAI and Anthropic) usually charge some storage fee for long caches but greatly reduce the input cost of repeated requests.

A mixed strategy: using the 12-hour cache to store shared instructions and the 30-minute cache to handle newer conversation content can balance reuse savings with avoiding long-cache fees for frequently changing parts.

Prompt caching can reuse computation results for long contexts that recur, significantly reducing duplicate billing.

Prompt caching typically has a short lifetime (on the order of minutes) and is keyed by provider, model, and content prefix.

After switching providers, the new provider's cache does not contain the previous prefix, and the entire context must be processed again at full price.

Major large-model providers (such as Anthropic and OpenAI) all support server-side caching of request prefixes, and the prefix portion of subsequent requests that exactly matches the cached content can be billed at a discount and need not be recomputed.

Prompt caching requires the request prefix to match byte for byte, and caches usually have a time to live (such as 5 minutes to 1 hour), after which they must be written again.

In RAG scenarios, the retrieved context changes with every query, and if it is placed early in the prompt it breaks cache hits for subsequent content, so the static system prompt is usually placed first to benefit from caching, with dynamic retrieved document chunks placed later.

Prompt caching reuses the already-computed intermediate state of the same long context submitted repeatedly (such as a fixed large document or system prompt), thereby reducing the billing cost for that portion of the input.

Prompt caching usually has an expiration period, requires rewriting after content changes, and the cache discount applies only to the input side, while output tokens are still billed at the standard rate.