Cache Hit
A discount mechanism in API billing whereby, when the prompt prefix of a request highly repeats that of recent requests, the server directly reuses already-processed content, thereby reducing billing costs.
Core Facts
Timeline (last 90 days)
Cache Hit means that when the context prefix of a request is exactly the same as that of a previous request, the server can directly reuse the already-computed KV cache without re-inference, so the unit price can be as low as two cents.
Some providers offer discounts on input tokens that hit the cache, but if the warm-up request itself misses or the probe is too short, the discount may not take effect.
Long system prompts also squeeze the context window and reduce cache hit efficiency.
Cache hit determination is based on token-by-token comparison of the prefix: only the part that is continuously identical starting from the first token counts as a hit, and once a difference appears, everything after it is counted as a miss.
In Agent scenarios, the system prompt, tool definitions and conversation history often make up the vast majority of the input, and the cache hit rate directly determines the actual bill; if a brand-new context is injected every turn, the cache benefit shrinks substantially and the cost ratio between the two models changes.
Noting constraints in the tool description (such as maximum length, cache freshness, and no duplicate fetching) can significantly reduce an Agent's number of repeat fetches.
Under the cache-hit billing mechanism, the portion that hits the cache is billed at a lower price; in Agent scenarios the cache hit rate is often as high as 90% or above, and the cache price has a far greater impact on the actual costs of heavy Agent users than the nominal input unit price.
A cache hit means the model server recognizes that the prefix of the current request highly overlaps with a previous request, and directly reuses the already-computed KV cache without re-running inference, thereby reducing the actual computation load.
Cache hit is a billing optimization mechanism in large language model APIs, with rates typically only 1/4 or even lower than that of regular input tokens.
Cache hits reduce the billing rate per request, not the absolute amount of tokens consumed.
2 more timeline events
All Facts (13)
The Prompt Caching mechanism can reuse the KV Cache of identical prefix contexts, typically reducing input token costs by 50%–90%.
80%UnverifiedCache Hit means that when the context prefix of a request is exactly the same as that of a previous request, the server can directly reuse the already-computed KV cache without re-inference, so the unit price can be as low as two cents.
50%UnverifiedSome providers offer discounts on input tokens that hit the cache, but if the warm-up request itself misses or the probe is too short, the discount may not take effect.
50%UnverifiedLong system prompts also squeeze the context window and reduce cache hit efficiency.
50%UnverifiedCache hit determination is based on token-by-token comparison of the prefix: only the part that is continuously identical starting from the first token counts as a hit, and once a difference appears, everything after it is counted as a miss.
50%UnverifiedIn Agent scenarios, the system prompt, tool definitions and conversation history often make up the vast majority of the input, and the cache hit rate directly determines the actual bill; if a brand-new context is injected every turn, the cache benefit shrinks substantially and the cost ratio between the two models changes.
50%UnverifiedNoting constraints in the tool description (such as maximum length, cache freshness, and no duplicate fetching) can significantly reduce an Agent's number of repeat fetches.
50%UnverifiedUnder the cache-hit billing mechanism, the portion that hits the cache is billed at a lower price; in Agent scenarios the cache hit rate is often as high as 90% or above, and the cache price has a far greater impact on the actual costs of heavy Agent users than the nominal input unit price.
50%UnverifiedA cache hit means the model server recognizes that the prefix of the current request highly overlaps with a previous request, and directly reuses the already-computed KV cache without re-running inference, thereby reducing the actual computation load.
50%UnverifiedCache hit is a billing optimization mechanism in large language model APIs, with rates typically only 1/4 or even lower than that of regular input tokens.
50%UnverifiedCache hits reduce the billing rate per request, not the absolute amount of tokens consumed.
50%UnverifiedSome platforms offer discounts of up to 75%–90% on input tokens that hit the cache.
50%UnverifiedOn some platforms, tokens that hit the cache still count toward the plan's quota consumption, calculated at a discounted rate rather than being truly free.
50%