LLM Response Caching in Practice: Building an Exact-Match Cache for OpenAI-Compatible APIs

In real production applications, a significant portion of LLM requests are completely identical — the same system prompt, the same user question, the same parameters — sent again simply because a page re-rendered, a test suite re-ran, a scheduled job retried, or two different users asked the same FAQ.

Aman Kumar, an OpenAI-compatible gateway operator, noticed this pattern in his traffic: every repeated request burns latency, and depending on how you're billed, money or quota as well. Adding a caching layer on the client side can eliminate most of this waste. This article walks through the approach he actually uses: what belongs in the cache key, where to store the data, how to handle streaming responses, and which scenarios you should never cache in the first place.

Client-Side Caching ≠ Provider-Side Prompt Caching

Many people conflate this with the "prompt caching" offered by some providers. Some vendors discount long, repeated prefixes (like a massive system prompt) at a reduced rate — that's genuinely useful, but it's a different feature: the model still runs, you still wait for generation, and the request still goes out.

Client-side response caching skips the request entirely. If the input is identical and you're fine receiving an identical response, a stored answer can be returned in milliseconds. The two approaches are complementary — provider-side caching handles requests that differ only at the end, while your own cache handles requests that don't change at all. Understanding this distinction is the prerequisite for avoiding duplicated effort.

Provider-side Prompt Caching (as implemented by OpenAI, Anthropic, etc.) is fundamentally an optimization at the KV Cache layer of model inference: for requests with identical prefixes, the inference engine can reuse previously computed attention key-value pairs, reducing redundant computation and offering a token discount (typically 50–75% off input token costs). But this optimization is transparent to the caller — the request still goes out, the network round-trip still happens, the model still runs inference on the remaining portion, and latency is still measured in seconds.

Client-side response caching operates at an entirely different layer: it makes its decision before the request ever leaves the client, and on a cache hit, latency can be as low as single-digit milliseconds. The two also differ in applicable granularity: KV Cache excels at "long system prompt + varying user questions" scenarios where prefixes repeat, while client-side caching only works for completely identical inputs. That's precisely why combining both covers a broader range of repetition patterns.

What Fields Must Be in the Cache Key

This is where most homegrown caches go wrong. If the cache key is missing anything that affects the output, you'll serve wrong answers. Following the author's approach, the key is a hash of:

• The base URL (so staging vs. production, or two different providers, never share entries)
• The model ID, exactly as sent
• The full messages array, including the system prompt
• temperature, top_p, max_tokens, stop, and seed if used
• tools, tool_choice, and response_format
• A version string that you increment whenever you change the prompt or post-processing logic

One critical security detail: the API key must never appear in plaintext in the cache key, nor be stored anywhere in the cache. If you need per-tenant isolation, use a tenant ID instead.

When serializing, sort the keys so that {"a":1,"b":2} and {"b":2,"a":1} hash to the same value:

import hashlib, json

CACHE_VERSION = "v3"

def cache_key(base_url: str, payload: dict) -> str:
    material = {
        "v": CACHE_VERSION,
        "base_url": base_url.rstrip("/"),
        **payload,
    }
    blob = json.dumps(material, sort_keys=True, ensure_ascii=False, separators=(",", ":"))
    return hashlib.sha256(blob.encode("utf-8")).hexdigest()

A Minimal Working SQLite Cache

For single-process or small services, SQLite is sufficient. It survives restarts and requires no extra infrastructure.

import sqlite3, time
from openai import OpenAI

client = OpenAI(base_url=BASE_URL, api_key="sk-your-key")

db = sqlite3.connect("llm_cache.sqlite3", check_same_thread=False)
db.execute("""CREATE TABLE IF NOT EXISTS cache (
    k TEXT PRIMARY KEY, body TEXT NOT NULL, created REAL NOT NULL)""")

TTL_SECONDS = 7 * 24 * 3600

def cached_chat(**payload) -> dict:
    payload.pop("stream", None)
    k = cache_key(BASE_URL, payload)
    row = db.execute("SELECT body, created FROM cache WHERE k=?", (k,)).fetchone()
    if row and time.time() - row[1] < TTL_SECONDS:
        return json.loads(row[0])
    resp = client.chat.completions.create(**payload)
    data = resp.model_dump()
    choice = data["choices"][0]
    if choice.get("finish_reason") in ("stop", "tool_calls"):
        db.execute("INSERT OR REPLACE INTO cache VALUES (?,?,?)",
                   (k, json.dumps(data), time.time()))
        db.commit()
    return data

There are two easily overlooked but important details here.

First, only store responses where finish_reason is stop or tool_calls. If the answer was truncated due to length, or something went wrong mid-generation, you don't want to freeze that in the cache for a week. Second, errors are never cached. A 429 or 5xx will raise an exception before the write happens, so the next call will retry — which is exactly the behavior you want.

Streaming Responses Can Be Cached Too

Streaming output and caching may seem contradictory, but they aren't. On a cache miss, you stream content to the user as usual while collecting all the chunks; once the stream ends cleanly, you concatenate the text and store it. On a cache hit, you can either return the full text at once, or replay it in small chunks to simulate the streaming experience the UI expects.

def cached_stream(**payload):
    payload.pop("stream", None)
    k = cache_key(BASE_URL, payload)
    row = db.execute("SELECT body FROM cache WHERE k=?", (k,)).fetchone()
    if row:
        text = json.loads(row[0])["choices"][0]["message"]["content"]
        for i in range(0, len(text), 40):
            yield text[i:i+40]
        return
    parts, finish = [], None
    for chunk in client.chat.completions.create(stream=True, **payload):
        if not chunk.choices:
            continue
        delta = chunk.choices[0].delta.content or ""
        finish = chunk.choices[0].finish_reason or finish
        parts.append(delta)
        yield delta
    if finish == "stop":
        body = {"choices": [{"message": {"role": "assistant",
                "content": "".join(parts)}, "finish_reason": "stop"}]}
        db.execute("INSERT OR REPLACE INTO cache VALUES (?,?,?)",
                   (k, json.dumps(body), time.time()))
        db.commit()

The elegant part: if the user disconnects mid-stream, the generator is closed, finish never becomes stop, and no incomplete content is written to the cache.

Temperature and "Should This Even Be Cached?"

Caching returns the same answer every time. When you're expecting the same answer, that's perfect; when variety is the whole point, it's wrong.

Good candidates for caching:

• Classification, labeling, information extraction, and routing at temperature=0
• Document summaries that won't change (use a content hash as the key, not the filename)
• FAQ-style responses and help center bots
• Batch pipeline tasks like "describe this item"
• Test suites and CI runs, where repeated calls are pure waste

Poor candidates for caching:

• Creative generation where users click "regenerate" expecting something new
• Tasks that depend on the current time, live data, or tool results fetched at request time
• Multi-turn conversations — the full history is almost never repeated verbatim (caching won't hurt, it just won't hit)

The author's rule of thumb: when temperature is 0 and the input is fully self-contained in the request, cache by default; in all other cases, enable caching explicitly and on demand.

Scaling Up with Redis

Once you have multiple workers, a per-process SQLite file can't be shared. At that point, swap the storage for Redis's SET key value EX ttl — the key function stays the same. Keep values compact (store only the fields you'll read back) and set a memory limit with an LRU eviction policy to prevent a wave of unique prompts from filling up your machine.

At this stage it's also worth adding request coalescing: if ten workers miss on the same key at the same moment, only one should call the API while the rest wait for its result. A short-lived Redis lock (SET lock:<key> 1 NX EX 30) is sufficient for most applications.

Request coalescing is essentially a concurrency control pattern that says "let one request do the work, and have everyone else wait and reuse its result" — also known as request deduplication or cache stampede protection. In high-concurrency scenarios, when a cache entry expires (or before it's first written), dozens of workers can discover a cache miss within milliseconds and simultaneously fire identical API requests upstream — this is a "cache stampede." The cost isn't just redundant spending; the sudden spike in concurrent requests can also trigger upstream rate limiting (429).

Using Redis's SET lock:<key> 1 NX EX 30 implements a lightweight distributed lock: NX (Not eXists) ensures only the first worker to execute the command acquires the lock, while the rest discover the lock already exists and enter a brief polling or wait loop; EX 30 sets a 30-second automatic expiry, preventing a dead lock if the lock-holding process crashes. For most LLM gateway scenarios, this granularity is more than adequate — LLM inference typically takes seconds to tens of seconds, well within the 30-second lock timeout.

How to Measure Whether Caching Is Worth It

For each feature, track three numbers: hit count, miss count, and latency for each. In the author's experience, internal tools and batch jobs tend to have high hit rates, while open-ended chat rarely hits. If a feature's hit rate is still close to zero after a week, remove the cache there — it's just dead code.

Also watch out for stale answers. If you change a prompt but forget to increment CACHE_VERSION, users will receive old-style responses for the entire TTL period. The author's solution: increment the version number in the same commit that changes the prompt, and keep TTLs on the order of days rather than months.

The Relationship to Pay-Per-Token and Monthly Plans

Under per-token billing, caching directly saves money. Under a "daily request quota" monthly plan, it saves quota — which matters especially for agents or batch jobs that would otherwise hit their limit before the day ends. Either way, the request you never sent is also the fastest request you'll ever make.

It's worth noting that the original author built an OpenAI-compatible gateway product, and the example base URLs come from that product. But all of the methods above apply equally to any OpenAI-compatible endpoint — just swap BASE_URL, the key, and the model ID for whatever your provider supplies.