One Misplaced Block Made Every Message Rewrite 19,000 Cached Tokens

Introduction: The Hidden Trap in Cache Optimization

In AI coding agents, prompt caching is the most cost-effective optimization available: system prompts and tool definitions are fixed across requests, so providers can cache them and read them back at a fraction of the price. Yet a QA check by the Atlarix team revealed that in Build mode, every message was rewriting the entire cache instead of reading from it. The root cause turned out to be a single misplaced block. This article takes a deep dive into how the cache bug was discovered, fixed, and what key lessons came out of it.

KV Cache refers to the Key and Value tensors that a model saves for already-processed tokens during autoregressive generation, avoiding the need to recompute the entire context for every new token. For long system prompts (often tens of thousands of tokens), this computation is substantial.

Prompt caching is essentially persisting the KV Cache to storage and loading it directly when a subsequent request hits the same prefix. This also explains why writing to the cache costs more than regular input—writing means additionally storing these tensors; reading from the cache is cheap because it skips the full forward pass over the prefix and only loads from storage. The cache lifetime (5 minutes or 1 hour) corresponds to the storage retention period, after which it must be rewritten.

Cache Mechanism in Brief: Exact Prefix Matching Is Key

Claude's caching is based on exact prefix matching: from tool definitions and system prompts through the conversation content, up to the marked breakpoint. If any byte before the breakpoint changes, everything from that byte onward is a cache miss and gets rewritten to the cache. According to Anthropic's pricing, cache writes cost more than regular input (1.25x for 5-minute entries, 2x for 1-hour entries), while cache reads cost only about one-tenth the price. Therefore, a cache that is written every time is worse than no cache at all.

Prefix Caching is an optimization technique in LLM inference services that trades space for time and cost. The principle: when multiple requests share the same prefix (such as system prompts, tool definitions, or few-shot examples), the server can reuse the previously computed KV Cache and skip redundant forward computation over the identical token sequence.

Mainstream implementations include Anthropic's Prompt Caching, OpenAI's automatic caching, and RadixAttention in inference frameworks like vLLM. Their common prerequisite is that the prefix must be token-for-token identical—change even one character and all cache from that position onward is invalidated. As a result, engineering practice typically splits prompts into a "stable prefix + dynamic suffix" and explicitly marks a cache breakpoint for the stable prefix. Understanding this is the foundation for judging whether caching is working.

What We Were Sending: The Misalignment Between Stable Head and Per-Turn Tail

Atlarix's system prompt is split into two parts: the stable head contains the agent's identity, tool rules, and mode instructions, unchanged throughout the conversation, and therefore carries the cache breakpoint with a 1-hour lifetime; the per-turn tail contains the current date and the request for this message, changes with every message, and should be sent as a separate block after the breakpoint.

Ask mode implemented this correctly, but Build mode merged the two parts into a single string and then appended context, guidelines, and project notes after it. As a result, the request text sat in the middle of the same block that carried the breakpoint, causing everything after the breakpoint to be rewritten every time.

Why It Went Unnoticed: The Blind Spot of Single-Message Checks

Within a single message, the agent makes multiple model calls to process tool results, and these calls share the same system prompt. So the logs showed healthy cache reads: on the second step of a message, the cache read 19,379 tokens. But the cache misses only occurred between messages, because the request text changed. Tests that checked messages one at a time always passed, masking the problem.

Agent Loop refers to the multiple rounds of model calls an AI agent makes to fulfill a single user request: the model outputs a tool call, the system executes the tool and returns the result, the model continues reasoning, and so on until it produces a final answer. All model calls triggered by the same user message share an identical system prompt and conversation history prefix, so the cache hit rate between them is naturally high.

This is exactly the mechanism that masked the problem: multiple calls within a single message show plenty of cache reads, and the logs look perfectly healthy; only when comparing across user messages does the cache miss caused by the dynamic portion of the system prompt surface. When debugging cache issues, you must distinguish between "intra-message calls" and "inter-message calls."

How We Found It: Comparing Two Real Requests

During QA on the Anthropic-format provider, the team captured two consecutive request bodies from the same conversation and compared them. The first difference appeared at character 22,966 of the 24,000-character system block: the <user_context> containing the user request. The content before and after was identical—only this one block differed. It was this misplaced block that caused the entire cache to be rewritten.

The Fix: Splitting the Cache Block and the Per-Turn Tail

The fix was to re-split the Build prompt into two parts before sending: the cached portion no longer contains anything that changes per message, and the per-turn tail is sent as a separate system block after it. Not a single word of the prompt itself changed—only the position of the tail was adjusted. Other parts of the app that read the prompt still read it in full.

A new test simulates the QA process: run two messages and require that the cache block be byte-for-byte identical and that the request be in a subsequent block. The test fails on the old code and passes on the new code.

Before and After: The Data Speaks

Using Claude Haiku 4.5 on Amazon Bedrock, across three messages in the same conversation, here are the cache read/write figures in Build mode:

Phase	Cache Reads	Cache Writes
Before fix, messages 2 and 3	0	~19,600 each
After fix, message 1	0	19,348
After fix, message 2	18,767	597
After fix, message 3	18,767	620

From the second message onward, the fixed portion of the prompt is now read from cache at about one-tenth the input price, whereas before it was written as a 1-hour cache at twice the input price. For the fixed portion of each message, this is about 15x cheaper on Claude, and the same layout lets any prefix-caching provider benefit.

The Core Lesson: Changing Content Must Come After the Breakpoint

The rule that came out of this incident: anything that changes per message must be placed after the last cache breakpoint, and the only reliable check is to compare two real requests—not a single request. The fix shipped with Atlarix version 15.9.

For developers building AI agents, this case is a warning: cache optimization isn't just about marking breakpoints correctly—you also need to ensure dynamic content is positioned correctly and verify it with byte-level comparisons across messages. Ignore this, and your costs may go up instead of down.

One-line Summary

A misplaced dynamic block caused every message to rewrite about 19K cached tokens; after the fix, costs dropped roughly 15x.