Prompt Caching Explained: The Key Technology for Saving Money with AI Agents

A Money-Saving Tool That's Often Misunderstood

Imagine you're using a coding Agent, and the conversation has accumulated 50,000 tokens of context. When you send a new 1,000-token question, the total context becomes 51,000 tokens. Here's the question: will you pay full price for all 51,000 tokens?

The answer is — if your agent harness is well-designed, most likely not. But if it's poorly designed, you'll pay full price for the entire context, and the costs will be staggering.

This is where Prompt Caching comes into play. It's a technology that can save you substantial costs when using large language models, but only if you understand and implement it correctly. This article breaks down how it works and the best practices for using it.

The Core Principle of Prompt Caching: It's the Input That Gets Cached, Not the Output

If you have a software development background, you might try to understand Prompt Caching through the lens of database caching. Traditional caching logic works like this: an application queries a database, the database stores the result in cache, and subsequent identical queries return the cached result without reprocessing.

Many people assume Prompt Caching works the same way — you send a prompt to the LLM, it returns a response that gets cached, and the next identical prompt returns the cached response directly. This is completely wrong.

Caching an LLM's output is virtually pointless. What actually gets cached is the input. Understanding this is the key to mastering Prompt Caching.

Why Cache Input Instead of Output: The KV Cache Under the Hood

The underlying implementation of Prompt Caching relies on the KV Cache (Key-Value Cache) mechanism in the Transformer architecture. In Transformer self-attention layers, each token generates corresponding Key and Value vectors used to compute attention weights. When the model processes a prefix of text it has seen before, it doesn't need to recompute these KV pairs — it loads them directly from cache, dramatically reducing GPU computation. This is why cache matching must start strictly from the prefix — once any token in the middle changes, the attention calculations for all subsequent tokens change, invalidating the cached KV pairs. Service providers maintain these KV Caches in GPU memory or high-speed storage and set TTL (Time To Live) values to manage cache lifecycle, automatically freeing memory resources upon expiration.

Why Agents Repeatedly Send the Same Tokens

Imagine your conversation with an Agent has reached 50,000 tokens, containing various messages, tool calls, return results, etc. When you ask a new question (say 1,000 tokens), the Agent must send all 51,000 tokens to the LLM for complete context understanding. The LLM might reply with 3,000 tokens. Then when you ask another question, you need to send all 55,000 tokens.

As you can see, we're sending the same tokens to the LLM over and over again, only appending a small amount of new content at the end each time — this is exactly how Agents work: append the next part of the conversation, then resend the entire history.

Modern Agent frameworks (like LangChain, AutoGen, Claude Code, etc.) essentially maintain a growing list of messages when interacting with LLMs. Each LLM call serializes and sends the complete message history — including system prompts, user messages, assistant replies, tool calls and their results. This "stateless" design means the LLM itself doesn't retain session state; all context memory is managed and transmitted by the client.

Even interfaces like OpenAI's Responses API, where the LLM appears to "remember" what was said before, are still reprocessing the entire context behind the scenes — they just hide the process. This also explains why maintaining extremely large contexts in coding sessions isn't wise — costs can spiral out of control quickly.

Without Prompt Caching, Costs Grow Exponentially

A so-called "50,000-token" session actually consumes far more than 50,000 tokens. The real token consumption is: 50k + 51k + 54k + 55k… and so on, quickly ballooning exponentially.

Token Billing Basics

LLM API billing is based on tokens — the smallest processing units that text is split into by a tokenizer. Different models use different tokenization strategies: the GPT series uses BPE (Byte Pair Encoding), where one English word typically corresponds to 1-3 tokens, while Chinese characters usually take 1-2 tokens each. APIs charge separately for input tokens and output tokens, with output tokens typically 3-4x more expensive than input tokens because the generation process requires autoregressive decoding token by token, resulting in higher computational density. Therefore, Prompt Caching primarily targets cost optimization on the input side.

If you're on a more expensive API like OpenAI, where input pricing is around $4 per million tokens, this repeated sending becomes extremely costly.

Take an Agent session of just 200,000 tokens as an example — the cost differences across Claude Opus, GPT-5, Gemini, Kimi, Grok, and DeepSeek are enormous. Without caching, a short 200k-token session on Opus and GPT costs around $41. But once caching is enabled, costs drop dramatically. DeepSeek is particularly "absurdly cheap," approaching nearly free.

Prompt Caching Pricing Rules and Prefix Matching

Today, virtually all LLM APIs use differentiated pricing: the first time the LLM "sees" a token, it charges full price. The second, third, and every subsequent time it sees identical content, the price drops significantly — typically to just 10% of the full price.

The critical point here is prefix matching: only consecutive sequences that are completely identical starting from the very first token can hit the cache. This means if your request is [system prompt + conversation history + new question], the system prompt and history portions (the prefix) can hit the cache, while the newly appended question is charged at full price. Different providers have different minimum cache granularity requirements: Anthropic requires at least 1024 tokens in the prefix to trigger caching, while OpenAI's threshold is 128 tokens. This also explains why the "append-only" pattern is so cache-friendly — the prefix of each request highly overlaps with the previous one.

Looking at the cost curve: without caching, session costs rise exponentially with context growth; with caching enabled, the curve remains roughly linear. Your goal is to hit that 10% discount price as much as possible.

Real-World Results: 10.9 Million Tokens for Just 6 Cents

In TAL (the Python port of Py), Prompt Caching is enabled for nearly all available inference providers. Here are the test results using DeepSeek v4 Flash (via Hugging Face inference provider):

The data shows: the entire session exchanged 10.9 million tokens with the LLM, while the context window is only 126K. Because tokens are sent back and forth every turn, the cumulative token count reached 10.9 million — with a final cost of just 6 cents.

In TAL's exported session report, the blue line represents cache hits and the red line represents full-price writes. At the beginning of the session, everything is charged at full price; as the conversation grows and the same tokens are repeatedly read, the cache hit rate gradually increases.

You might not have noticed that caches expire: the two points in the graph where costs "spike back up" correspond to lunch and dinner breaks — the cache expired due to timeout, and re-reading required paying full price again. Different providers have different expiration times: OpenAI is about 1 hour, Anthropic defaults to 5 minutes (1 hour with Claude Code authentication). Cache expiration is essentially a trade-off providers make between GPU memory pressure and user experience — memory is a scarce resource, and maintaining KV Caches for large numbers of users long-term consumes precious inference capacity.

Best Practices for Prompt Caching in Agent Design

To maximize your Agent's use of Prompt Caching, keep the following points in mind.

Pay Attention to Cache Expiration Times

The first thing to confirm is the cache expiration time for your inference provider. On Hugging Face inference providers, TAL routes all requests to the same provider, hitting the "warm" cache and almost never missing any cached content. This routing strategy is especially important in distributed inference scenarios — if requests are load-balanced across different GPU nodes, each node's KV Cache is independent, and cache hit rates drop significantly.

Confirm Whether Your Provider Automatically Enables Caching

Some providers automatically cache inputs (like OpenAI and Hugging Face inference providers), but others (like Anthropic or Gemini) do not enable caching automatically — you need to manually enable it in your API calls. For Anthropic, for example, you need to add a cache_control field to your messages to mark cache breakpoints, telling the API which content should be cached. While this explicit marking increases development complexity, it also gives developers finer-grained control — you can precisely specify where cache boundaries should be.

Avoid Using Dynamic System Prompts

This point is critically important.

Suppose your system prompt is 10,000 tokens and the session grows to 200,000 tokens. These tokens could be cached on every request. But if your system prompt contains timestamps, current working directories, or other changing content, the moment it changes, it invalidates the entire cache for everything that follows it.

The reason behind this is directly related to prefix matching: the system prompt sits at the very front of the entire request — it's the first part of the prefix. Once any single token in the prefix changes, all KV Cache entries from that position onward become invalid, even if the subsequent conversation history hasn't changed at all. It's like dominoes — push over the first one, and everything after it falls.

Therefore, never include any dynamic content in your system prompt, including timestamps, working directories that might change mid-session, dynamically updated tool lists, etc. — otherwise session costs will skyrocket. The correct approach is to place dynamic information at the end of the message sequence (such as in the latest user message), or use a separate system message at the end of the conversation to supplement context like time.

Understand That Conversation Compression Resets the Cache

When you compress a long conversation by summarizing it into a brief digest, this also invalidates the cache. This is normal behavior of the compression mechanism, not a bug, but you should be aware that it resets the cache.

Conversation compression is a common strategy for context window management: when conversation length approaches the model's context window limit (e.g., 128K tokens), the framework summarizes early conversations into a concise digest, replacing the original detailed messages. While this effectively controls the token count per request, since the compressed text is completely different from the original prefix, all cached KV pairs are invalidated. Trade-off strategies include: delaying compression as long as possible to maximize cache utilization, using a sliding window to preserve the most recent N turns of original conversation, or actively warming the new cache through consecutive requests after compression.

Summary

Prompt Caching is an essential consideration when building cost-effective Agent systems. The core principles can be summarized as:

• System prompts and history should not change dynamically during a conversation
• Conversation history should remain "append-only" to avoid small mistakes that invalidate the cache
• Pay attention to cache expiration times — if there's sufficient time remaining, consider extending cache validity for specific providers
• Use Agent frameworks that support cache hit rate monitoring (like Py or TAL) to observe cache hit rates in real time

For developers who work with coding Agents daily, correctly implementing Prompt Caching can reduce bills from tens of dollars to just a few cents, and is a foundational capability for building sustainable Agent systems. As Agent systems evolve toward longer sessions and more complex tool chains, understanding and optimizing caching mechanisms will become the critical dividing line between "it works" and "it works affordably."