Prompt Cache Deep Dive: From KV Cache to Agent Engineering Practice

When working with large model APIs, you may have noticed an interesting phenomenon: the same model, the same tokens — yet the price can differ by one or two orders of magnitude. The key behind this is Prompt Cache. Current mainstream model pricing follows a strikingly consistent pattern — cache writes cost roughly 1.25× the standard price, while cache reads cost only about 0.1×. This enormous price gap is the entry point for understanding the cost structure of LLM inference.

This article systematically walks through the complete technical chain from KV Cache and PagedAttention to Agent engineering practice, based on an in-depth technical breakdown of Prompt Cache.

Four-Layer Architecture: Understanding Prompt Cache, KV Cache, and PagedAttention

Prompt Cache, Prefix Cache, KV Cache, and PagedAttention often appear together, but they don't belong to the same layer. Understanding them bottom-up helps avoid confusion.

The Complete Chain from Infrastructure to Product

• PagedAttention (bottom layer): Handles GPU memory management. It maps logically contiguous KV sequences to a set of discrete physical blocks, handling allocation, sharing, and reclamation.
• KV Cache: The actual state the model saves — the Key and Value tensors from each Attention layer.
• Prefix Cache: Determines under what conditions KV state can be reused across requests. It searches by token prefix and stops at the first mismatch.
• Prompt Cache (product layer): What you see in API docs and billing — automatic caching, explicit hot paths, TTL, billing rules, and so on.

The key insight is that what model providers are selling you is essentially a reuse mechanism for prefix computation — it doesn't necessarily guarantee which underlying open-source implementation is being used. To understand these four layers, just keep four questions in mind: what system are you buying into, how is a cache hit determined, what information is reused on a hit, and how is that state persisted in GPU memory.

Prefill and Decode: Where LLM Inference Latency Comes From

A single request can be roughly split into two phases: Prefill and Decode.

The Essential Difference Between the Two Phases

During Prefill, the entire input prompt is received, and the GPU can process these input tokens in parallel — performing one large round of matrix computation to build up the internal state needed for subsequent generation. The user sees nothing during this time, waiting for the first token — so Prefill is a major component of Time to First Token (TTFT).

Once the first token is output, the model enters the Decode phase, where each step adds one or a small batch of positions, reads through the historical KV, and predicts the next token. The real-time token-by-token output you see is TPOT (Time Per Output Token).

Total latency is roughly: TTFT + number of subsequent tokens × per-step decode time.

Prompt Cache operates specifically on the Prefill computation — it eliminates redundant prefix computation. However, it's worth noting that Decode still has to happen; there's no way around that. Additionally, real-world TTFT includes queuing, routing, network, and batching overhead that caching doesn't eliminate.

KV Cache: Trading GPU Memory for Computation

In each Attention layer, the hidden state is projected through three weight matrices to produce Q, K, and V. When generating the next token, the new Query computes relevance scores against all previous Keys, then aggregates all Values by those weights.

The key insight: the historical Keys and Values are fixed — appending a new token doesn't change them. Since they don't change, there's no need to recompute them from scratch at every step. So the Q from the first pass is discarded after use, while K and V are retained in the cache. The next step only computes new Q, K, V for the new position — the new Q reads the full historical KV, and the new KV is appended to the end.

How KV Cache Memory Capacity Is Calculated

KV Cache capacity is calculated as: 2 tensors (K and V) × number of layers × number of tokens × number of KV heads × head dimension × bytes per data type — growing linearly with context length. The PagedAttention paper uses OPT-13B in FP16 as an example: each token requires roughly 800 KB, so 2,048 tokens consume about 1.6 GB. This is precisely why techniques like GQA and MQA are now widely adopted to compress this overhead.

PagedAttention: OS-Inspired Paging for GPU Memory Management

The early approach was to pre-allocate a single contiguous memory block for each sequence — but since output length is unknown in advance, this leads to internal waste and significant external fragmentation.

PagedAttention directly borrows the OS paging model: logical KV entries (L0, L1, L2, L3) appear contiguous, but the underlying Block Table maps them to discrete physical blocks.

Two Core Capabilities of PagedAttention

First, blocks are allocated incrementally in fixed sizes, eliminating the need to reserve space for fragmentation. Second, for multiple sequences that share a common prefix, the Block Table can point to the same physical blocks, with reference counting managing their lifetime. For example, sequences A and B may share certain physical blocks — reads require no copying; only when one sequence needs to modify a shared block does the system allocate a new block via Copy-on-Write.

It's worth emphasizing: PagedAttention is fundamentally a KV memory management mechanism. Prefix Cache is a separate concern at a different layer.

Cache Hit Conditions: More Than Just Matching Text

A common pitfall: two documents that look identical to the human eye don't guarantee cache reuse.

Consider three requests: A and B have completely identical token sequences from system prompt to document — their state can be shared. But C, despite containing the same document, has a different context prefix before it. Because in a Transformer, a position's hidden state depends on all preceding tokens and is also affected by positional encoding, the KV computed for C is entirely different from A and B.

Multiple Dimensions That Affect Computational Identity

Production-environment matching also depends on many other dimensions: tokenizer, chat template, model and weight version, raw multimodal inputs, and so on — all of which together constitute the "computational identity." Self-hosted inference engines typically match on token IDs plus identity information; hosted APIs generally require the serialized prefix to be byte-for-byte identical.

The hit condition in one sentence: completely identical token prefix + completely identical computational identity.

Prefix Hashing and RadixTree: Two Reuse Implementations in vLLM and SGLang

vLLM's Automatic Prefix Caching uses a cross-block hash chain: tokens are divided into fixed-size blocks, the hash of the first block is derived from its own tokens plus additional metadata, and each subsequent block includes the previous block's hash — forming a chain. A block can only obtain a reusable identity if all content before it is also identical. As of vLLM 0.11, SHA-256 is used by default to reduce collision and information leakage risk, though this is no substitute for tenant isolation — when isolation is required, it still depends on unpredictable salts or physical separation. Idle cache entries enter LRU and are evicted from the tail only when memory pressure builds.

SGLang's RadixAttention organizes the KV corresponding to prompts and generated outputs into a RadixTree: in multi-turn conversation scenarios, common prefixes sit at the top of the tree, and paths branch naturally where messages diverge. New requests traverse the tree to find the longest matching prefix. According to the paper, under workloads with no prefix reuse, the overhead of this structure is less than 0.3% (under specific experimental conditions).

Five Causes of Prompt Cache Misses and How to Diagnose Them

Cache misses typically come down to a few bottlenecks:

1. Shared prefix is too short: If it's too short, it may not meet the hosted model's minimum caching token threshold — overseas providers generally require thousands of tokens, while DeepSeek has relatively smaller cache blocks.
2. Computational identity mismatch: Timestamps, request IDs, JSON key ordering, tokenizer, chat template, model version — any one of these changing will cause the hit to terminate at the point of divergence.
3. Cache has been evicted: TTL expiration, LRU eviction, or memory pressure can all turn a hot path cold again.
4. Routing miss: If the request is routed to a machine that doesn't hold the cached state, it has to recompute regardless.
5. Decode dominates: When output is very long, Decode dominates the total time, and prefix cache hits provide limited end-to-end speedup.

For diagnosis, you should track cache reads/writes, uncached input tokens, TTFT for hits vs. misses, routing targets, and Decode ratio together — not just single-request hit rate.

Agent Engineering Practice: Organizing Stable Cached Contexts

A tool-calling Agent is naturally a loop, and naturally cache-friendly — as long as the token sequence of prior content stays stable, the vast majority of the previous turn can become the cached prefix for the next, with only the new Tool Result needing to be re-prefilled.

The Stability Spectrum: More Stable Content Goes First

When designing prompts, arrange all content along a stability spectrum:

• Front: Tool definitions, system rules, fixed few-shot examples (rarely change)
• Middle: Knowledge base snapshots (change by version)
• End: Conversation history, real-time retrieval results, Tool Results, user questions (change most frequently)

Problems almost always originate from application-layer "casual modifications": someone inserts the current timestamp or a random request ID at the beginning, someone dynamically adds or removes tools, someone lets a JSON serializer decide its own schema ordering — what looks like a few-line change invalidates a long stretch of cached context downstream. A better approach is to append as new messages instead, and switch tool sets and templates by version.

Security and Isolation Trade-offs in Cache Sharing

Cache sharing also introduces privacy risks: timing differences are observable, and an attacker who can repeatedly probe whether a sensitive prefix results in lower TTFT could infer whether that content has been used by someone else. vLLM supports mixing a cache salt into the first block's hash, and recommends using an unpredictable 256-bit random value rather than a username or account ID. The finer-grained the isolation, the lower the cross-user sharing rate — this is an explicit trade-off that must be consciously accepted.

Summary: A Decision Framework for Deploying Prompt Cache

The value of Prompt Cache lies in safely skipping repeated Prefill computation — but it comes with costs in GPU memory, write overhead, eviction, routing, versioning, and privacy. Whether caching is worthwhile depends on three questions: is the shared prefix long enough, can the reuse frequency justify the cost, and does it cross a system boundary (if so, namespace and tenant isolation must come first).

The rollout sequence is also clear: measure first, then reorder, then define identity and isolation scope, then continuously test after launch. There's no need to force-cache every request.

Understanding the underlying implementation of Prompt Cache not only helps you read LLM billing statements, but also guides context organization in Agent systems — ensuring that old prefixes within each lifecycle are reused predictably and stably, rather than left unmanaged.