HotKV Semantic Cache Field Test: The Truth Behind 70% Cost Reduction on Claude Tokens

A Rigorous but Limited Benchmark Test

The HotKV team ran an end-to-end test on the Banking77 dataset: they sent 3,000 customer service questions to four Claude models, once directly to the models (retaining Anthropic prompt caching), and once with a HotKV cache layer in front of the models. The results were eye-catching — model calls, input/output tokens, and API costs all dropped by about 70%.

But the most valuable part of this report is precisely its candor about applicability boundaries. HotKV itself emphasizes: this is not a universal "save 70%" promise, but a measurement result under a specific workload.

How the Cache Actually Works

HotKV performs two lookups before calling the model. The first is exact matching: it computes a SHA-256 hash of the normalized question, model identifier, and system prompt version, and returns directly on a hit. The second is semantic matching: it generates vectors using OpenAI's text-embedding-3-small and finds questions in the cache whose cosine similarity exceeds a threshold. Only when both miss does it actually call the model and write the answer back to the cache.

The semantic threshold was fixed at 0.81 before testing. This value was calibrated on the Banking77 training set — the lowest threshold guaranteeing that at least 98% of semantic hits point to the same intent. But on the actual test set, the intent consistency rate of semantic hits was 96.7% to 96.8%, with each model having 30 to 31 hits on the wrong intent. This shows that accuracy at the intent-label level can no longer be fully equated with answer correctness.

Cosine similarity measures how close two vectors are in direction, ranging from -1 to 1, with values closer to 1 indicating greater semantic similarity between two texts. After a semantic cache maps each question to a high-dimensional vector, it relies on this value to judge whether a new question is "equivalent" to an old one in the cache. A threshold of 0.81 means the cache is only hit when similarity reaches 0.81 or above. It is important to understand that high cosine similarity only indicates close vector direction and does not guarantee that answers are interchangeable — this is precisely the mathematical root of the "near-miss question" false-hit risk mentioned in the article: the vectors for "deposit" and "withdrawal" may be quite close, yet their intents are diametrically opposed.

text-embedding-3-small is an embedding model provided by OpenAI, responsible for converting text into fixed-dimension numerical vectors for similarity retrieval. It is small, low-cost to call, and has short latency, so it is often used in scenarios like semantic caching and retrieval augmentation that require "vectorizing every request." However, embedding models only capture the overall direction of semantics and have limited ability to distinguish individual key details (such as amounts, directional words, negations), which explains why semantic cache hit judgments can be inaccurate on subtly altered questions.

Where the Savings Come From

Across the four models (Fable 5.1, Opus 5.5, Sonnet 5.5, Haiku 5.5), total call count dropped from 12,000 to 3,588, tokens dropped 70.1%, and cost dropped 69.6%. About 70% of requests were answered by the cache, with exact matching contributing about 39% and the semantic layer adding about 31%.

The key insight is: most of the savings come from exact duplication of questions, not semantic similarity. If the semantic layer were removed, exact matching alone would require at most 1,264 model calls (corresponding to 1,264 distinct questions), already saving about 58%. The semantic layer contributed only about 12 additional percentage points on this workload — and it is precisely the source of answer quality risk.

Another easily misread number is the token composition. Of the counted tokens, 90% to 93% are cache reads of the same 2,200-token system prompt, which Anthropic charges very little for. Output tokens account for only 6% to 9% of total tokens, yet 73% to 92% of baseline cost. So the main driver of cost reduction is eliminating complete calls, not cosmetic games with token counts.

Prompt caching is a mechanism offered by vendors such as Anthropic: when multiple requests share the same long prefix (such as a 2,200-token system prompt), the model can cache and reuse the computation results for that prefix, billing the cached-read portion at a price far below regular input. The fact that 90% to 93% of the token counts in the article are cache reads of the system prompt is exactly this mechanism at work. Understanding this is crucial: it shows that "70% fewer tokens" and "70% lower cost" are two different things, and the main driver of cost reduction is eliminating entire model calls, not bookkeeping changes in per-token pricing.

Speed Gains and Their Cost

Median response time dropped from 1.4 to 2.9 seconds down to about 125 to 128 milliseconds, an 11x to 23x improvement. But this is the overall median, because 70% of requests going through the cache pulls the median into the semantic-hit range (about 130 milliseconds, mainly the embedding API call). The median latency of an exact hit itself is about 0.05 milliseconds, but this is a local loopback measurement, not a network environment.

Long-tail improvement is limited: 30% of requests still call the model, and P99 latency on Fable 5.1 dropped from 6.6 seconds to 6.4 seconds, barely changing. For latency-sensitive applications, caching can only address duplicate and similar requests; it cannot eliminate cold starts or long-tail latency.

Answer Quality: The Real Weak Spot

This is the most honest and most noteworthy part of the entire report. HotKV had a different Claude model judge each semantic hit: the judge was shown the user question, the cached answer, and the baseline answer, and asked to determine which one actually answered the question.

At the 0.81 threshold, the proportion of cached answers that "fit the question" was 5.8 to 10.7 percentage points lower than fresh answers. Converted to all responses (semantic hits are about 31%), the overall quality loss is about 2 to 3 percentage points. Haiku 5.5 had the largest gap at 10.7 percentage points, with a cached-answer pass rate of only 67.9%.

Raising the threshold can mitigate the problem. Scanning multiple thresholds on Sonnet 5.5 found that above 0.88, cached answers no longer showed a quality disadvantage while still saving 64% of cost; 0.90 saved 61%. But the evidence is weak: at 0.90 the 95% confidence interval for the difference ranged from 4 points worse to 7 points better, and at 0.95 there were only 50 hits, with each row judged on a different subset.

More severe is the "near-miss question" test. HotKV had Opus 5.5 generate 119 question pairs, each consisting of a real question plus a version with only one key detail changed (for example, deposit vs. withdrawal). At the 0.81 threshold, 62% of the altered questions falsely hit; even raising it to 0.90, 25% still hit. Meanwhile, the hit rate for true synonymous rewrites dropped from 39% to 4%. This means semantic caching inherently faces a trade-off: the higher the threshold, the safer, but the fewer rewrites it can cover.

How to Read These Numbers

Savings are highly dependent on duplication rate. Changing the intent distribution from Zipf-skewed to uniform dropped the cost reduction for Sonnet and Haiku from 70% to 56%; if all 3,000 questions were distinct, the reduction would be only 38%, all from semantic matching of different questions under the same intent. For a workload with no duplicate or similar questions at all, savings are zero.

Extrapolating to scale also requires caution. Based on 1 million requests per day over 30 days per month, Fable 5.1 could save about $211,000 per month, while Haiku 5.5 would save only about $1,900. This extrapolation is arithmetic based on the measured per-request cost, not another actual test, and it excludes HotKV's own hosting and operational costs.

The test environment itself also has limitations: each condition was run only once, on a single quad-core server, with the client and cache on the same machine, and the cache was not warmed up (cache hit rate was about 52% for the first 1,000 requests and about 81% for the last 1,000).

Zipf distribution (Zipf's law) describes a long-tail phenomenon of "a few high-frequency items and a large number of low-frequency items": the frequency of the item ranked n is roughly proportional to 1/n. Real customer service question distributions closely match this shape — a few common questions are asked repeatedly, while many questions rarely appear, so the cache can cover most traffic. The article uses a "uniform distribution" as a control, essentially artificially eliminating this head-concentration effect to quantify "how much of the savings actually comes from question duplication." This also reminds developers: before evaluating the benefits of semantic caching, you should first measure the duplication rate distribution of your own traffic, rather than directly applying the 70% figure.

Practical Advice for Developers

The greatest value of this benchmark test is that it clearly delineates the applicability boundaries of semantic caching. For high-duplication, intent-converging scenarios like customer service and FAQs, the economics of caching are very favorable, but you need to accept some answer quality loss or raise the threshold above 0.88 to buy safety. For scenarios where questions almost never repeat, or where subtle wording differences lead to completely different answers (financial transaction instructions, medical dosages, etc.), the risk of semantic caching may outweigh the benefits, and exact matching or direct model calls are safer.

HotKV's exact-match cache is included in all plans, while semantic caching (SCACHE.*) requires the Performance plan or above. The team recommends validating answer quality on your own questions before deciding whether to enable semantic reuse — this advice itself is more worth remembering than the 70% figure.

One-line Summary

HotKV benchmark shows semantic caching can save about 70% of cost, but quality risk and benefits are highly dependent on question duplication rate.

Paragraph Summary

This article interprets HotKV's end-to-end cache benchmark test on four Claude models using the Banking77 dataset. The test shows model calls, tokens, and API costs all dropped by about 70%, but the report candidly admits this is a measurement result under a specific workload rather than a universal promise. The key finding is that most savings come from exact duplication of questions (exact matching alone already saves about 58%), while the semantic layer contributes only about 12 additional percentage points yet carries almost all the answer quality risk: at the 0.81 threshold, cached-answer pass rates are 5.8 to 10.7 percentage points lower, and 62% of "subtly altered questions" falsely hit. The article also discusses long-tail latency, the near-miss question trade-off, the decisive impact of duplication rate on savings, and applicability boundary advice for developers.