Prompt Caching in Practice: Put Stable Content First and Stop Paying Twice for the Same Tokens

Most LLM applications send the same few thousand tokens on every request: tool definitions, system prompts, reference documents, a handful of examples. The part that actually changes is usually just the last few hundred tokens — the user's actual question. Without caching, you pay full input price to reprocess that identical prefix, over and over again.

Prompt caching exists precisely to solve this problem — but only if you structure your prompts in the right order. The single most important principle to internalize is: put stable content first, put variable content last.

What Prompt Caching Actually Caches

When a model reads your prompt, it computes intermediate key-value (KV) states for every token. Prompt caching saves those states for reusable prefixes — the unchanging tokens at the start of your prompt. If your next request starts with the exact same prefix, the provider reuses the saved computation and only processes the new portion.

The operative phrase is exact same. Both OpenAI and Anthropic match against the fully rendered prefix. Change even a single token at the top, and everything after it is a cache miss.

Think of it like a bookmark in a thick book: the model can jump back to where it left off, but only if nobody rewrote the earlier pages.

The KV cache is a core component of the attention mechanism in Transformer architectures. As the model processes each token, it generates Key and Value vectors used to compute attention weights between that token and prior context. In autoregressive generation, these vectors could theoretically be reused — but without caching, every new request recomputes the full prefix KV states from scratch, resulting in massive redundant computation. Prompt caching persists these computation results in the provider's inference infrastructure (not on the client side), allowing subsequent requests to skip that computation entirely when the prefix matches. This also explains why a minimum token threshold exists (1,024 tokens for OpenAI): for short prefixes, the overhead of cache lookup and management outweighs the savings, making caching counterproductive.

Why This Is Worth Your Attention

The numbers from official provider documentation are compelling:

• Anthropic: Cache reads are billed at 0.1× the base input price on most Claude models. Writing to the default 5-minute cache costs 1.25×, and a 1-hour cache costs 2×. A 5-minute cache refreshes for free each time it's used.
• OpenAI: Caching is enabled by default on supported models. Starting with GPT-5.6, cache writes cost 1.25× and reads cost 0.1× the uncached input rate, with a minimum cacheable prefix of 1,024 tokens. OpenAI's official example: one write plus nine full reads totals 2.15× the prefix input price — versus 10× without caching.

Beyond cost savings, caching also reduces time-to-first-token because the model skips reprocessing the long prefix.

The Ordering That Makes Caching Work

Every request should be structured in this order:

1. Tools and system prompt. These almost never change. On Anthropic, the explicit cache hierarchy is tools first, then system, then messages.
2. Shared reference material. Documents, policy terms, few-shot examples, or a long contract you're querying against.
3. Conversation history. Append new turns; don't rewrite old ones.
4. New user message and any dynamic content. Timestamps, usernames, per-request context.

The logic behind this ordering is straightforward: the less frequently something changes, the earlier it should appear — and the higher the probability of a cache hit.

The Mistake Almost Everyone Makes

The most common mistake is putting dynamic content at the top. The classic example is a system prompt that opens with a line like Current date and time: 2026-10-12 09:04:31. That line changes on every request, so the prefix never matches and you never get a cache hit.

Anthropic's documentation describes an even worse variant of the same trap: if you place cache_control on a block that contains a timestamp and the user's message, you'll trigger a fresh cache write on every single request while never reading from cache. Move the breakpoint to the last block that stays unchanged, and every subsequent request will read from cache instead.

The documentation also lists several other silent cache-busters:

• Changing tool names, descriptions, schemas, or their order.
• Some languages randomize JSON key ordering when serializing tool calls (Anthropic specifically calls out Swift and Go).
• Toggling settings that get rendered into the prompt, such as tool_choice, images, or reasoning/thinking settings.
• Summarizing or compressing conversation history, which rewrites the prefix.

JSON key order randomization is an engineering detail worth treating with special caution. In Python, dict has guaranteed insertion order since 3.7, and json.dumps output is stable. But in some standard library implementations in other languages, serializing a map or struct to JSON does not guarantee key order. Anthropic's SDK documentation calls out Swift (the Encodable protocol doesn't guarantee ordering) and Go (encoding/json serializes maps in alphabetical key order, but struct field order depends on the definition and can vary across versions or reflection calls). The practical fix: use a custom encoder that guarantees serialization order, or use ordered data structures when building tool definitions — and before going to production, compare the raw payloads of two identical requests byte-for-byte to confirm your tool definitions are consistent.

What This Looks Like in Code

Anthropic's approach uses an explicit breakpoint at the end of the stable section:

response = client.messages.create(
    model="<your-claude-model>",
    max_tokens=1024,
    system=[
        {"type": "text", "text": "You are a support assistant for Acme..."},
        {
            "type": "text",
            "text": "<full product docs and policy here>",
            "cache_control": {"type": "ephemeral"},  # cache up to this point
        },
    ],
    messages=[
        {"role": "user", "content": f"(sent at {now}) {user_question}"},  # dynamic content, after the breakpoint
    ],
)
print(response.usage)  # check cache_read_input_tokens / cache_creation_input_tokens

OpenAI requires no code changes for basic caching on supported models. Keep the prefix stable and check usage.input_tokens_details.cached_tokens to confirm it's working.

Verify It — Don't Assume It

Cache failures are silent. A prompt that falls below the minimum token threshold simply won't be cached, with no error or warning. You have to measure:

• Anthropic: Watch cache_read_input_tokens and cache_creation_input_tokens. If both are zero, nothing was cached.
• OpenAI: Watch usage.input_tokens_details.cached_tokens and the Prompt Caching dashboard.

A useful ongoing practice: track a simple ratio each day — cached tokens divided by total input tokens. If it drops sharply after a deployment, someone probably slipped dynamic content into the prefix.

Quick Checklist

• Tools and system prompt come first, identical on every call
• Documents and examples come next
• User input, timestamps, and per-user personalization data come last
• Never reorder tools or let JSON key ordering get shuffled
• Place the breakpoint on the last block that never changes
• After every prompt change, glance at the cache token fields

Before you spend a week trimming your prompt to cut costs, first confirm you're actually getting cache hits. Reordering is usually the cheaper fix.