Prompt Caching

The core principle of prompt caching is cross-request reuse of the KV Cache, where already-computed KV states can be cached and reused when multiple requests share the same prefix.

The KV Cache mechanism avoids repeated computation of historical tokens, but its memory usage grows linearly with context length.

In Anthropic's prompt caching mechanism, cache-hit tokens are charged at only 10% of the standard price.

OpenAI Prompt Caching can reduce the cost of repeated prefixes to 10%–25% of the original price.

The general principle for achieving cache hits is to keep the prefix of the request context as stable as possible, placing fixed content such as system prompts and tool definitions at the very front, with per-turn user input appended at the end.

Prompt caching saves pre-computed KV matrices on the server side, reducing the processing of repeated prefixes from O(n²) to O(1) cache reads, enabling up to 90% compression in latency and cost.

Cache entries have a validity period of approximately 5 minutes (default) to 1 hour (extendable), after which they are treated as cache misses.

The condition for an implicit cache hit is that the content, model, and parameters are exactly identical, and the stable context must be at the front of the prompt, otherwise there is no hit.

The fix for the cache point not moving forward is to place the cache point on the last content block of the newest message, regardless of its role.

When running on Amazon Bedrock, prompt cache invalidation produces no error or exceptional status code and the request still returns 200, but it causes input tokens to be recomputed at full price plus an extra 25% cache write premium.

Bedrock's prompt cache looks back to old cache entries at a distance of about 20 content blocks, and the author measured the boundary between 21 and 22 content blocks: when 21 new blocks are added it still hits, while with 22 new blocks it misses every time.

Prompt caching reuses input processing, not model output or memory; each request is still a stateless, independent inference.

Prompt caching, memory, compression, and retrieval are four mechanisms that are often confused but solve different problems: prompt caching reuses input processing, memory preserves information across interactions, compression reduces the context carried, and retrieval selects relevant information.

Major providers such as Anthropic's Claude and OpenAI's GPT-4o allow caching of fixed prefix portions of requests, so that when the cache is hit, those tokens are not recomputed, reducing latency and cost.

Prompt caching is the highest-impact cost lever, with OpenAI's pricing sheet showing that cached inputs can save up to 90% in costs.

Prompt caching is based on prefix matching, reusing already-computed KV Cache to skip redundant computation, so fixed content at the beginning of prompts should be kept consistent to maximize cache hit rate.

Prompt caching requires an exact token sequence match as a prerequisite; any change at any position in the prefix invalidates the cache, and all tokens from that changed position onward must be recomputed.

GLM 5.3 supports prompt caching on Bedrock to optimize cost and latency.

Amazon Bedrock's prompt caching implementation performs hash matching on cached prefixes server-side, requiring developers only to mark the content they wish to cache.

Cache-safe reasoning changes mean that developers can adjust reasoning behavior while retaining the cost advantages provided by prompt caching.