[KongchangAI]
· 4 min read· 2,172 words

Semantic Caching in Practice: Speeding Up AI Systems with Python and Redis

Semantic Caching in Practice: Speeding Up AI Systems with Python and Redis

Implement semantic caching in Python and Redis to reuse historical answers via vector similarity and speed up AI systems.

The article starts from the limitations of exact caching and explains how semantic caching maps queries to vectors via embeddings and reuses historical answers through similarity matching. The demo first hand-writes the core logic with NumPy: maintaining parallel lists of historical embeddings and answers, computing similarities in batch with matrix multiplication, taking the best match with argmax, and emphasizing writing only when producing original answers to avoid cache pollution. It then focuses on threshold tuning, noting that 0.6 and 0.7 misjudge semantically opposite or differently intentioned queries, and 0.9 is a safer default. It then introduces three high-risk scenarios (context-dependent conversations, environment-dependent queries, personal and time-sensitive data) and three engineering measures—TTL, eviction, and scoping—and finally builds a persistent, cross-process production-grade solution with Redis VL's SemanticCache, noting that its distance_threshold is a distance, not a similarity.

Semantic Caching in Practice: Speeding Up AI Systems with Python and Redis

When an AI system handles massive volumes of user requests every day, what really slows it down is often not the model itself, but the fact that repeated or highly similar questions are recomputed from scratch over and over again. Semantic Caching is the key technique that solves this problem: it embeds user queries into vector space, and as long as a new question is sufficiently similar to a historical one, it directly reuses the existing result and skips the entire inference pipeline.

This article is based on a hands-on demonstration by an AI engineer, starting from the most naive dictionary cache and gradually moving to a production-grade solution built on Redis, with a focus on unpacking the easily overlooked pitfalls of threshold tuning, TTL, eviction policies, and scoping.

From Exact Caching to Semantic Caching: What's the Difference

Traditional caching can only hit exactly identical strings. When a user asks "What is the capital of France?", the result can only be reused the next time someone asks the exact same question word for word. This is indeed useful in some systems—there are always people who repeat the same question verbatim—but the coverage is very narrow.

Semantic caching takes a different approach: it first converts a query into a vector via an embedding model, then compares its similarity against historical vectors in the cache. If the similarity exceeds a set threshold, the two questions are considered to "mean the same thing" and the cached answer is returned directly. "What is the capital of France", "I would like to know the capital of France", and "I want to know the capital of France" will be close together in vector space, so they can all hit the same cache entry.

Semantic caching compares query similarity via embeddings

In the demo, a baseline script without caching was first run against five questions, with "What is dynamic programming" intentionally repeated twice. The results showed that each request took anywhere from just over 2 to just over 4 seconds, and the two answers to the same question were not worded exactly the same—because each was a fresh inference.

Embeddings and vector similarity are the foundation for understanding semantic caching. An embedding model (such as OpenAI's text-embedding-3-small) maps text into a high-dimensional dense vector (typically hundreds to over a thousand dimensions), and texts with similar semantics are closer together in vector space. The common metric for measuring the similarity of two vectors is cosine similarity, i.e., the cosine of the angle between them, ranging from -1 to 1, where values closer to 1 indicate more aligned directions and more similar semantics. In practice, vectors are usually normalized first, and a dot product is used instead of the cosine computation for efficiency. The demo uses NumPy matrix multiplication to compute the similarity between the current vector and all historical vectors in one shot, precisely leveraging the property that the dot product of normalized vectors is equivalent to cosine similarity.

The Python Implementation of Exact Caching and Its Ceiling

Before getting into semantic caching, the demo first implements exact caching with a dictionary plus SHA-256 hashing. The role of the hash is to compress a query of arbitrary length into a fixed-length key, which is more uniform than using the raw string directly as a key.

The effect is very intuitive: the first "What is dynamic programming" runs normal inference, the second hits the cache, and the time drops from seconds to nearly instantaneous (printed as a tiny scientific-notation value), with the returned text exactly identical to the first time.

The problem is that, apart from that one exactly duplicated query, all the other similar questions phrased differently were misses. "Can you explain dynamic programming" and "Explain what dynamic programming is" are almost semantically equivalent, yet exact caching is powerless. This is why embeddings must be introduced.

Hand-Writing the Core Logic of Semantic Caching with NumPy

The implementation of semantic caching is not complicated; the core is to maintain two parallel lists: one storing historical embeddings and one storing the corresponding answers, linked by index.

Each time a new query comes in, it first obtains a vector from OpenAI's text-embedding-3-small model, then converts the historical embeddings into a NumPy array and performs matrix multiplication with the current vector to compute all similarities at once. np.argmax is used to get the index of the most similar record, and then the corresponding similarity value is read.

The decision logic is: if the similarity exceeds the threshold (set to 0.9 in the demo), return the cached answer; otherwise, run full inference and append the new embedding and answer to the two lists respectively.

There is one key detail in practice: only write to the cache when producing an original answer. If a cache hit also writes the reused result back, the cache will be continuously "polluted"—each new question may be sufficiently similar to the previous one, propagating layer by layer, and eventually completely different queries could be matched to the wrong answers.

After getting it running, the effect is: the first question takes about 2.9 seconds on cold start, the second fully duplicated question has a similarity of about 1 (printed as a value close to 1, with a tiny numerical error) and the time drops to 0.14 seconds. "Can you explain dynamic programming" has a similarity of 0.933, and "Explain what dynamic programming is" has a similarity of 0.92, both hitting the cache. Finally, "What is recursion" has a similarity of only 0.39, so it goes through normal inference, taking about 2 seconds.

The risk of cache amplification deserves a separate mention. Semantic caching differs from exact caching in that its hit determination is inherently fuzzy: if every hit writes the reused result together with the new query's embedding back into the cache, a chain of "semantic drift" forms—A is similar to B, B is similar to C, but A may already be far from C. As the chain lengthens, more and more entries deviating from the original intent accumulate in the cache, eventually matching irrelevant queries to wrong answers. Therefore, the write operation must be tied to "whether the model was actually invoked": only when a cold start produces an original answer should the current query embedding and answer be stored, while the hit path is read-only. This is also the most basic discipline for maintaining cache quality.

Threshold Tuning: 0.9 Is a Good Starting Point, but Not a Panacea

The threshold is the parameter in semantic caching that most needs repeated experimentation. The demo specifically wrote a tuning script that uses a set of "trap questions" to test hits under different thresholds, including variants that should hit (such as "explain dynamic programming") and similar questions that should never hit (such as "don't explain dynamic programming", "what is linear programming", "what is dynamic typing").

Hits and false positives under different thresholds

The test results are very telling:

  • 0.6: Too loose. "don't explain dynamic programming" was judged a hit, even though its semantics are actually negative; "when should I use dynamic programming" also hit, but returned "what is dynamic programming" rather than "when to use it".
  • 0.7: Equally bad. "dynamic programming explained", which should have hit, was instead a miss, while the two wrong matches still existed.
  • 0.8 and 0.9: The main difference is in the "when should I use dynamic programming" entry; 0.9 is stricter and behaves more as expected.

The demonstrator's experience is: 0.9 is a good default, depending on how strict the business requirements are for accuracy. A lower threshold gives higher recall but greater risk of wrong answers; a higher one is safer but reduces the cache benefit.

Three Types of Caching Pitfalls That Must Be Avoided

Semantic caching is not a universal accelerator; in some scenarios, using it actually produces wrong answers. The demo explicitly points out several high-risk situations.

Context-dependent conversations. Follow-up questions like "tell me more about this" or "what do you mean by this" have correct answers that depend entirely on the preceding text. If they are cached, subsequent users asking the same words under a completely different topic will get irrelevant answers.

Environment-dependent queries. A Windows user and a Linux user asking "how do I install something on my system" should get different answers. Caching without distinguishing the environment is bound to go wrong.

Personal data and time-sensitive content. Queries like "What's on my calendar" are both bound to a specific user and change over time. Even if the same question is asked again ten minutes later, the answer should be updated.

TTL, Eviction, and Scoping: Three Engineering Measures

To address the pitfalls above, the demo presents three corresponding engineering measures.

TTL (Time to Live)

Setting an expiration time for cache entries solves the timeliness problem. In the demo, TTL was set to 5 seconds, and the same question was asked three times in a row at 3-second intervals: the first time computes on cold start, the second hits the cache, and the third is judged expired because more than 5 seconds have passed since the first write, so it recomputes. TTL checking is essentially a time check on each read to confirm whether the entry is still "fresh".

Eviction

Limiting cache capacity, squeezing out the oldest entries when full. In the demo, the maximum capacity was set to 2: after asking "what is dynamic programming" and "what is recursion" in sequence, the cache was full, and when "what is a hash map" was asked, the oldest dynamic programming entry was evicted. Asking about dynamic programming afterward would then miss, while recursion and hash map remained in the cache.

Scope

Isolating the cache by scope to prevent different contexts from polluting each other. The two scopes in the demo are "answer in one sentence" and "answer in one sentence in German"—the same question uses different embedding pools and cache dictionaries under different scopes. In this way, asking in German will not hit the cache under the English scope, and vice versa.

Embedding cache structure isolated by scope

Building a Near-Production Solution with Redis VL

All the previous implementations are based on dictionaries and lists within the script, and the cache is gone once the process exits. The demo finally uses Redis VL (Redis Vector Library) to show a production-grade approach.

After installing redis-vl, import the SemanticCache class from redisvl.extensions.cache.llm and OpenAITextVectorizer from redisvl.utils.vectorize. Initialization requires a Redis connection address (the demo starts Redis 8 locally on port 6379 via Docker), a distance_threshold (note that this is a distance, not a similarity; a similarity threshold of 0.9 corresponds to a distance upper bound of 0.1), a TTL, and a vectorizer instance.

The read/write interface is very concise: use cache.check() to check whether there is a matching historical prompt; on a hit, take the response field from the first element of the returned result; on a miss, call the model normally and then use cache.store() to write the prompt and response into Redis.

The results after switching to Redis are exactly the same as the previous list implementation—miss, hit, hit, hit, miss—showing that the semantic matching logic has not changed, only that the storage has been switched from an in-memory list to a vector database that is persistent and shareable across processes.

Redis VL's SemanticCache relies on Redis's vector retrieval capabilities (such as the vector index of the Redis Query Engine) under the hood. It stores prompt vectors in the index and performs a nearest-neighbor (KNN) search with distance threshold filtering at query time. This is fundamentally different from the "full matrix multiplication" of the hand-written list implementation: the list approach has to compare against all historical vectors every time, with complexity growing linearly with cache size; whereas the vector index uses approximate nearest neighbor (ANN) algorithms to keep retrieval latency low even with massive numbers of entries, while also supporting persistence, multi-process sharing, and horizontal scaling. Another easily confused point is the distance metric: Redis VL uses cosine distance (1 minus cosine similarity) by default, so a similarity of 0.9 corresponds to a distance of 0.1, and the comparison direction of the threshold is opposite to that of the hand-written version. Special care is needed during configuration, otherwise you may encounter the counterintuitive phenomenon that "raising the threshold actually makes it looser".

When Is Semantic Caching Worth Using

The demonstrator's judgment on applicable scenarios is quite restrained. Suitable cases include: document retrieval, internal support bots, and search engines for large product catalogs—these scenarios have high query repetition rates and low dependence on context.

Unsuitable cases are agent conversations and highly context-dependent interactions. In other words, semantic caching is a sharp tool in an AI engineer's toolbox, but only on the premise that you can judge whether your business form matches. Used correctly, it can compress the response time of a large number of repeated requests from seconds to milliseconds; used incorrectly, it will only quietly return wrong answers.

#semantic caching#Redis VL#vector similarity#embedding#Python#AI system optimization#TTL eviction
Share:

Related articles