Prompt Caching: The API Cost Killer 90% of Developers Overlook

There's a nearly free discount hiding in your LLM API bill: for input tokens that have already been processed, providers charge only one-tenth the price. This is Prompt Caching, the highest-leverage cost optimization available on modern LLM APIs. Yet most developers either don't use it or use it wrong, resulting in dismal cache hit rates.

The Underlying Logic of Prompt Caching: Stable Prefixes Are the Key

When a provider processes a request, it feeds the input tokens into the model for computation—this is where most of your money goes. But if the next request begins with an identical token sequence, such as the same system prompt, the same codebase context, or the same set of few-shot examples, the provider doesn't need to recompute it. It directly reuses the cached intermediate results and charges only a fraction of the price.

The keyword here is stable prefix. Caching matches from the beginning of the prompt forward; the matched portion receives the discount. Once tokens diverge—a new user message, the latest tool return value—everything after that is billed at full price.

This mechanism imposes a clear architectural constraint: stable content first, variable content last. System instructions, repository context, examples, reference documents—put them all at the top, byte-for-byte identical on every call. Put changing content at the end. Get the order right, and the cache handles the rest.

KV Cache (key-value cache) is the technical foundation that makes prompt caching possible. During autoregressive generation, a Transformer model computes Key and Value vectors for every processed token at each attention layer and stores them in GPU memory for subsequent tokens to query. Without caching, every request would recompute these tensors for the entire input sequence, with compute cost proportional to input length. Prompt caching essentially preserves the KV tensors from a previous request, indexed by prefix, so a cache hit can load them directly and skip the forward computation. This also explains two constraints: first, the cache must match contiguously from the start of the sequence, because attention is causal—later tokens depend on the KV of all preceding positions; second, the byte-for-byte requirement stems from the tokenizer—even one extra space changes the tokenization, breaking the positional correspondence of the KV tensors. Different providers' cache lifetimes (typically ranging from minutes to hours) and storage granularity are also determined by this GPU memory/storage trade-off.

Doing the Math: How Much Does a 90% Discount Actually Save?

Take GPT-5.6 Sol's public pricing as an example: fresh input costs $4.00 per million tokens, while cache reads cost only $0.40 per million tokens—exactly the inverse of a 90% discount, meaning you pay only one-tenth.

Plugging in a reference workload: 1,100 tasks per month, each with 12K fresh input, 48K cached input, and 6K output tokens. Per-task cost: $0.0480 for fresh input, $0.0192 for cached input, $0.1200 for output. Note that 48K cached tokens is four times the fresh token volume, yet costs less than half as much. Without caching, those 48K tokens at full price would cost $0.1920, saving roughly $190 per month across 1,100 tasks—the bulk of your input bill simply disappears.

One cost is often overlooked: cache writes. When a prefix is cached for the first time, the provider charges a premium—1.25x the input rate on GPT-5.6 Sol. In other words, the first call with a new prefix costs slightly more than no caching at all. But every subsequent reuse is nearly free. As long as you make more than a handful of calls, the math is overwhelmingly favorable—just don't model the first call as discounted.

Five Engineering Rules for Cache Hits

Cache misses are usually not the provider's fault—they're a prompt structure problem.

Fixed order, consistent every time. The stable prefix must be byte-for-byte identical—not just semantically the same, but the same tokens in the same order. Generate prompts programmatically from a fixed template, not by concatenating strings in whatever order the code happens to run.

No dynamic content in the prefix. Timestamps, request IDs, usernames, "today's date"—if any of these land in the cached region, every request has a different prefix and the cache never hits. Move dynamic content after the stable block, or accept that caching only starts after it.

Put the longest stable content first. Sort by change frequency: system prompt (never changes) → reference documents (rarely change) → conversation summary (changes slowly) → real-time request (changes every time). This maximizes the cached portion even when later blocks rotate.

Don't scramble tool definitions. In agent loops, tool schemas are often large stable blocks—perfect cache material. But if the framework reorders or regenerates them on every call, the cache breaks. Pin them down.

Verify, don't assume. After refactoring, check the usage dashboard for the cached token ratio. For repetitive workloads, aim for 80%+ of input tokens hitting the cache. If it's only 20%, something in the prefix is still changing—grab two consecutive prompts and diff them to find it.

Realistic Hit Rate Expectations

How high a hit rate you can achieve depends on the workload shape. Coding agents with stable repository context can realistically target 70% to 90% of input tokens cached—the system prompt and codebase context barely change between calls, only the task description and tool results vary. Chatbots with a fixed system prompt land at 50% to 80%—the system prompt caches well, but conversation history keeps growing and shifting, eroding the hit rate in long sessions; periodically summarizing history rather than resending it verbatim helps. One-off or highly variable tasks get only 0% to 30%—if every request's input is genuinely different, such as document processing or unique queries, there's little repeated prefix to cache. Don't force it; caching is built for repetition.

The pattern is clear: the more your workload resembles "same setup, different questions," the higher the hit rate; the more it resembles "everything different," the lower.

Why conversation history erodes cache hit rates deserves its own explanation. In multi-turn conversations, each new user message or assistant reply extends the history by some amount, and the cache can only match up to the last stable breakpoint. If the system prompt and early history are fully cached, the newly added turn is billed as fresh tokens—that's unavoidable. But the problem is that many implementations insert rolling summaries, current-turn metadata, or reformatted history near the front, causing the entire prefix to shift. An even more common trap is regenerating the full messages array and adjusting role markers every turn, introducing subtle differences into what should be identical. Mitigations include: keep history append-only without rewriting, pin periodic summaries as a separate block at the end of the stable region rather than the beginning, and avoid backfilling volatile fields like timestamps into history. A hit rate that declines as sessions grow longer is normal; the key is to ensure the decline comes from genuinely new content, not structural prefix drift.

Which Workloads Benefit Most

Biggest winners: agent coding loops (huge stable context, thousands of repetitions), RAG systems with fixed document sets, chatbots with long system prompts, evaluation frameworks that run the same prompt against many inputs. These also happen to be the scenarios with the largest input volumes—which is why caching matters so much.

Moderate gains: mixed workloads with partial repetition, where the system prompt caches but not much else. Still worth doing; configuration cost is nearly zero.

Skip entirely: genuinely one-off inference, batch jobs over unique documents, scenarios where inputs never repeat. The cache-write premium means you'll pay more for zero benefit.

How to Debug Cache Misses

If the hit rate is still low after refactoring, the culprit is almost always something small and dynamic hiding in the stable prefix. The debugging method isn't fancy but it works: grab two consecutive prompts, diff them, and see exactly which tokens differ. Common suspects include: template-injected timestamps and dates ("Current date: ..." at the top of the system prompt is a classic cache killer), session or request IDs inserted into instructions, tool definitions regenerated in non-deterministic order on each call (sort them, or better, serialize once and reuse), conversation history placed before rather than after the system prompt (every new message shifts the entire prefix), and whitespace/formatting differences from string templates rendering through different code paths. Fix one at a time and check the dashboard after each change—the hit rate should climb in steps. A plateau means you've found all the varying factors.

One more subtlety: minimum cacheable length. Providers typically require the stable prefix to exceed a minimum token count before caching kicks in, so short prompts may not qualify no matter how well structured. If your prompt is only a few hundred tokens, caching may simply not apply—and that's fine; there wasn't much to save on small inputs anyway. The value of caching shows up with large, repetitive contexts.

The Bottom Line

Prompt caching is the closest thing to "free money" in LLM APIs: roughly a 90% discount (really paying one-tenth) on your largest token category, at the cost of just ordering your prompt correctly. Developers paying full price for repetitive context aren't being overcharged—they're just sending the same tokens in a slightly different order every time. Fix the order, watch the dashboard, and watch the input line on your invoice collapse.

Background Notes

Tool schemas (Function Calling definitions) are typically sent with each request in agent loops as JSON Schema, and the model uses them to decide which tool to call and with what parameters. Providers like Anthropic and OpenAI include them as part of the system-level input that participates in prefix matching. The problem is that many agent frameworks organize tool lists using dictionaries or sets, and in languages like Python, dictionary iteration order and JSON serialization key ordering can vary across versions or code paths; some frameworks also dynamically inject the currently available subset of tools on each call. Both practices cause the tool definition block to drift byte-wise between turns, invalidating a stable block that could have cached tens of thousands of tokens. The safe approach is to serialize tool definitions into a fixed string at process startup (e.g., sorted by name, then json.dumps) and reuse the same one for the entire session; if tools genuinely need to be added or removed dynamically, split the changing part out after the stable block.