[KongchangAI]
ConceptCache Hit / 缓存命中定价

Cache Hit

A discount mechanism in API billing whereby, when the prompt prefix of a request highly repeats that of recent requests, the server directly reuses already-processed content, thereby reducing billing costs.

Core Facts

Timeline (last 90 days)

Nov 18

Cache Hit means that when the context prefix of a request is exactly the same as that of a previous request, the server can directly reuse the already-computed KV cache without re-inference, so the unit price can be as low as two cents.

Unverified50%
Nov 10

Some providers offer discounts on input tokens that hit the cache, but if the warm-up request itself misses or the probe is too short, the discount may not take effect.

Unverified50%
Nov 10

Long system prompts also squeeze the context window and reduce cache hit efficiency.

Unverified50%
Nov 6

Cache hit determination is based on token-by-token comparison of the prefix: only the part that is continuously identical starting from the first token counts as a hit, and once a difference appears, everything after it is counted as a miss.

Unverified50%
Oct 19

In Agent scenarios, the system prompt, tool definitions and conversation history often make up the vast majority of the input, and the cache hit rate directly determines the actual bill; if a brand-new context is injected every turn, the cache benefit shrinks substantially and the cost ratio between the two models changes.

Unverified50%
Oct 18

Noting constraints in the tool description (such as maximum length, cache freshness, and no duplicate fetching) can significantly reduce an Agent's number of repeat fetches.

Unverified50%
Sep 23

Under the cache-hit billing mechanism, the portion that hits the cache is billed at a lower price; in Agent scenarios the cache hit rate is often as high as 90% or above, and the cache price has a far greater impact on the actual costs of heavy Agent users than the nominal input unit price.

Unverified50%
Sep 21

A cache hit means the model server recognizes that the prefix of the current request highly overlaps with a previous request, and directly reuses the already-computed KV cache without re-running inference, thereby reducing the actual computation load.

Unverified50%
Sep 18

Cache hit is a billing optimization mechanism in large language model APIs, with rates typically only 1/4 or even lower than that of regular input tokens.

Unverified50%
Sep 17

Cache hits reduce the billing rate per request, not the absolute amount of tokens consumed.

Unverified50%

2 more timeline events

All Facts (13)

Verified

The Prompt Caching mechanism can reuse the KV Cache of identical prefix contexts, typically reducing input token costs by 50%–90%.

80%
Unverified

Cache Hit means that when the context prefix of a request is exactly the same as that of a previous request, the server can directly reuse the already-computed KV cache without re-inference, so the unit price can be as low as two cents.

50%
Unverified

Some providers offer discounts on input tokens that hit the cache, but if the warm-up request itself misses or the probe is too short, the discount may not take effect.

50%
Unverified

Long system prompts also squeeze the context window and reduce cache hit efficiency.

50%
Unverified

Cache hit determination is based on token-by-token comparison of the prefix: only the part that is continuously identical starting from the first token counts as a hit, and once a difference appears, everything after it is counted as a miss.

50%
Unverified

In Agent scenarios, the system prompt, tool definitions and conversation history often make up the vast majority of the input, and the cache hit rate directly determines the actual bill; if a brand-new context is injected every turn, the cache benefit shrinks substantially and the cost ratio between the two models changes.

50%
Unverified

Noting constraints in the tool description (such as maximum length, cache freshness, and no duplicate fetching) can significantly reduce an Agent's number of repeat fetches.

50%
Unverified

Under the cache-hit billing mechanism, the portion that hits the cache is billed at a lower price; in Agent scenarios the cache hit rate is often as high as 90% or above, and the cache price has a far greater impact on the actual costs of heavy Agent users than the nominal input unit price.

50%
Unverified

A cache hit means the model server recognizes that the prefix of the current request highly overlaps with a previous request, and directly reuses the already-computed KV cache without re-running inference, thereby reducing the actual computation load.

50%
Unverified

Cache hit is a billing optimization mechanism in large language model APIs, with rates typically only 1/4 or even lower than that of regular input tokens.

50%
Unverified

Cache hits reduce the billing rate per request, not the absolute amount of tokens consumed.

50%
Unverified

Some platforms offer discounts of up to 75%–90% on input tokens that hit the cache.

50%
Unverified

On some platforms, tokens that hit the cache still count toward the plan's quota consumption, calculated at a discounted rate rather than being truly free.

50%

Source Articles