What Is Prompt Caching? Principles, Use Cases, and Engineering Pitfalls

When asked about Prompt Caching in an interview, many people can only give a surface-level answer: "caching the prompt to save compute and speed things up." Once the interviewer follows up with "Is it the same as KV cache?" "What exactly is being cached?" "Which scenarios can use it, and what pitfalls are easy to hit?" quite a few candidates freeze on the spot.

In a video explaining real LLM interview questions, Bilibili creator Peng Yu broke down this high-frequency question from concept to implementation to pitfalls. Below, drawing on his explanation, we lay out the core logic of Prompt Caching.

Prompt Caching and KV Cache Are Not the Same Thing at All

The most easily confused issue is the relationship between Prompt Caching and regular KV cache. According to Peng Yu's analysis, these two are not even on the same dimension.

Regular KV cache is an optimization within a single request. In the same conversation, the KV generated from historical messages does not need to be recomputed; only the newly input content is incrementally computed. It serves incremental decoding within a single conversation.

Prompt Caching, by contrast, is an optimization across requests, across users, and across conversations. As long as the prompt prefixes of different requests are exactly the same, the corresponding KV results can be shared globally, without every user having to recompute from scratch.

One saves compute within a single conversation; the other reuses compute across massive numbers of requests. The order of magnitude of the benefit is completely different.

Why Must It Be Prefix Matching?

The answer lies in Transformer's causal attention mechanism. Each input token passes through the self-attention layer to generate corresponding Key and Value vectors, and causal attention has a key rule: the KV of each token depends only on all the tokens before it, not on the content after it.

Therefore, as long as the prefixes of two prompts are exactly the same, the generated KV is exactly the same, and can naturally be reused directly.

The implementation logic is not complicated: first hash the input prompt, match it against the cached KV prefix, and if there is a hit, skip this portion of Transformer computation and directly use the ready-made KV; only the differing suffix portion goes through inference computation to generate new KV and append it to the cache.

Causal Attention is the core mechanism of the Transformer decoder and the foundation that enables Decoder-only architectures (such as the GPT series) to generate text token by token. It adds a lower-triangular mask during self-attention computation: the token at position (i) can only "see" tokens at positions (0) through (i), and cannot see later content. This guarantees that for the (i)-th token, the Key and Value vectors it generates depend only on the prefix ([0, i]) and are unrelated to subsequent tokens. It is precisely this "only look backward" property that makes two prompts with exactly the same prefix produce exactly the same KV at the same positions, becoming the theoretical premise for Prompt Caching to safely reuse KV. Conversely, if the model uses bidirectional attention (such as the BERT encoder), the representation of each token is affected by the entire text, and prefix KV cannot be independently reused. This is also the fundamental reason Prompt Caching mainly appears in generative large models rather than encoder models.

Prefix Tree / Trie is a common data structure used by open-source inference engines for finer-grained KV reuse. Compared with simple whole-prefix hash matching, a prefix tree organizes cached KV into a tree by token sequence: each node corresponds to a common token prefix, and branches correspond to different subsequent content. As long as multiple requests share a common prefix of any length, they can hit the corresponding node in the tree and reuse the KV along that path, without requiring the two prompt prefixes to be exactly identical as a whole. This structure can significantly improve cache hit rates in multi-user and mixed-template scenarios, but it also brings complexity in node management, memory fragmentation, and eviction strategies (such as LRU combined with reference counting). vLLM's Automatic Prefix Caching and SGLang's RadixAttention are both engineering implementations of this idea.

Two Industry Implementation Paths

At present, Prompt Caching is mainly implemented along two paths in the industry:

On the closed-source API side, mainstream large model providers basically all natively support it, using exact prefix matching. Tokens that hit the cache are usually billed at only 10% to 30% of the original price, saving substantial costs in long-context scenarios.

On the open-source framework side, mainstream inference engines such as vLLM and SGLang all implement cross-request KV reuse, and some also support finer-grained prefix tree matching, together with cache eviction strategies such as LRU and TTL to balance GPU memory usage and hit rate.

Core Value: Save Time and Money

The value of Prompt Caching is concentrated in two points.

Significantly reduce time to first token. In long-prompt scenarios such as long-document Q&A and Agents with complex tools, once the prefix hits the cache, computation over thousands or even tens of thousands of tokens can be skipped. Time to first token can usually improve by 70% to 90%, and the improvement in interactive experience is very intuitive.

Significantly reduce inference cost. For model providers, a single GPU can handle more requests, directly improving throughput; for API callers, the cached portion is billed less, and overall costs in high-frequency scenarios such as templated calls and RAG can drop by more than half.

Anyone who has done batch inference should have this intuition: in long-prompt tasks, the bulk of compute lies in the common part. Caching is equivalent to spreading this fixed cost across countless requests, and the cost-effectiveness immediately stands out.

Four Typical Implementation Scenarios

Mentioning the following scenarios in an interview can directly demonstrate implementation experience:

1. Conversational services with fixed system prompts: role settings, output specifications, and safety rules are uniformly placed at the front globally, hitting the cache almost 100% of the time.
2. RAG knowledge base Q&A: document content goes in the prefix, the user question goes at the end, the document portion is cached directly, and only the user's new question is computed each time.
3. Tool-calling Agents: a fixed tool list and calling format are placed at the front, naturally suitable for caching.
4. Batch standardized processing: in scenarios such as batch content moderation, translation, and information classification, instruction templates are completely identical, and the template portion can be fully cached.

Four Pitfalls That Are Easy to Hit

This part is the deep-water zone interviewers love to probe, and only by answering it can you show real understanding.

Matching requirements are strict. Mainstream solutions all use character-level exact matching. One extra space, a changed punctuation mark, or an adjusted paragraph order will directly cause a miss.

Implementation requires proper prompt standardization governance. When many teams first launch caching, the hit rate is pitifully low, most likely because they did not standardize and wrote prompts too casually.

Only prefix matching is supported. Only repeated content placed at the very front can be reused; repetition in the middle or suffix is useless. Engineering practice should follow the principle of "fixed content first, dynamic content last."

GPU memory usage needs trade-offs. The cache is stored in GPU memory, and unlimited caching will crowd out available inference memory, instead hurting concurrency. A reasonable hot/cold eviction strategy must accompany it.

Semantic caching should be used with caution. The industry has tried using semantic similarity matching to improve hit rates, but it easily leads to attention drift and degraded output quality. At present, industrial implementation still mainly relies on exact matching.

LRU and TTL are two common cache eviction strategies that directly determine the balance between memory usage and hit rate. LRU (Least Recently Used) evicts by access time: cache blocks not hit for a period are released first, suitable for scenarios with clear hot/cold access distribution. TTL (Time To Live) sets a validity period for each cache block and reclaims it after expiration, suitable for scenarios with frequent content updates and low reuse value for stale data. Inference engines often combine the two, along with reference counting—cache blocks currently in use by a request cannot be evicted. Improper selection leads to two types of problems: too aggressive eviction causes the hit rate to plummet and the cache becomes useless; too conservative eviction fills up GPU memory and instead squeezes the available space for concurrent inference. Engineering practice requires tuning parameters based on the reuse frequency of prompt templates and the pace of document updates.

Semantic Caching means not requiring character-level exact consistency, but instead using vector similarity to determine that two prompts "mean something similar" and then reusing the result. Theoretically it can improve hit rate, but the risk is high: first, semantic similarity does not mean KV can be reused. Under causal attention, different token sequences naturally generate different KV, and forcing reuse will cause attention distribution drift and degraded output quality; second, it may misclassify requests that should be treated differently as the same, bringing safety and correctness issues. Therefore, the mainstream industry still insists on exact prefix matching, and semantic caching remains mostly at the exploration stage.

Time To First Token (TTFT) is one of the most critical experience metrics for large model inference, referring to the time from sending a request to receiving the first output token. It is mainly determined by the "prefill" stage—the model first needs to encode the entire input prompt into KV. The longer the prompt, the greater the prefill computation and the higher the TTFT. What Prompt Caching skips when it hits is exactly this prefill computation, so the improvement in TTFT is especially significant for long-prompt scenarios such as long-document Q&A and Agents. Users can intuitively feel "instant reply" instead of "stuck for ages."

Interview Answering Approach

Stringing the above together forms a progressive answer logic:

First give the core definition—it is a cross-request KV reuse optimization; then distinguish it from regular single-request KV cache; next explain the underlying principles and mainstream implementations; then give several implementation scenarios; finally supplement limitations and engineering considerations.

With this logic, the interviewer can tell that you are not just memorizing concepts, but truly understanding and thinking.