HotKV

HotKV's exact-match cache is included in all plans, while the semantic cache (SCACHE.*) requires the Performance plan or above.

HotKV's caching mechanism first does exact matching (SHA-256 hashing of the normalized question, model identifier, and system prompt version), then semantic matching (using OpenAI text-embedding-3-small to generate vectors, with a hit when cosine similarity exceeds the threshold), and only when neither hits does it call the model and write back to the cache.

Before testing, the semantic threshold was fixed at 0.81, a value calibrated on the Banking77 training set to guarantee that at least 98% of semantic hits point to the same intent; but on the actual test set the intent agreement rate of semantic hits was 96.7% to 96.8%, with each model having 30 to 31 hits on the wrong intent.

Across the four models, total calls dropped from 12000 to 3588, tokens dropped 70.1%, and cost dropped 69.6%; about 70% of requests were answered by the cache, of which exact matching contributed about 39% and the semantic layer contributed an additional about 31%.

Without the semantic layer, exact matching alone would require at most 1264 model calls, already saving about 58% of cost, with the semantic layer contributing only an additional about 12 percentage points on this workload.

P99 latency on Fable 5.1 dropped from 6.6 seconds to 6.4 seconds, a limited long-tail improvement, because 30% of requests still call the model.

At the 0.81 threshold, the proportion of cached answers that "fit the question" is 5.8 to 10.7 percentage points lower than fresh answers; converted to all responses, the overall quality loss is about 2 to 3 percentage points; Haiku 5.5 has the largest gap (10.7 percentage points), with a cached-answer pass rate of only 67.9%.

Scanning thresholds on Sonnet 5.5 found that above 0.88 cached answers no longer show a quality disadvantage while still saving 64% of cost; 0.90 saves 61%.

In a test of 119 pairs of "near-identical questions", at the 0.81 threshold 62% of modified questions incorrectly hit; even raised to 0.90, 25% still hit; the hit rate for genuine synonymous rewrites dropped from 39% to 4%.

Changing the intent distribution from Zipf-skewed to uniform, the cost reduction for Sonnet and Haiku drops from 70% to 56%; if all 3000 questions are different, the reduction is only 38%.

HotKV runs end-to-end tests on the BankING77 dataset against four Claude models (Fable 5.1, Opus 5.5, Sonnet 5.5, Haiku 5.5), and with HotKV caching added, model calls, input and output tokens, and API cost all drop by about 70%.

The test environment was a single four-core server, with the client and cache on the same machine, the cache not warmed up (cache hit rate about 52% for the first 1000 requests and about 81% for the last 1000), and each condition run only once.

Extrapolated at 1 million requests per day over 30 days per month, Fable 5.1 could save about $211,000 per month while Haiku 5.5 saves only about $1,900; this extrapolation is arithmetic based on the measured per-request cost, not another actual measurement, and excludes HotKV's own hosting and operations costs.