Build a Semantic Caching Proxy with FastAPI to Cut LLM Bills by 58%

The Problem: Fragile Commercial Gateways Are Costing You

There's an uncomfortable truth hiding inside production LLM deployments: in enterprise SaaS applications, 30% to 60% of LLM queries are duplicate or lightly paraphrased content. Customer service conversations, repetitive agent tasks, automated data enrichment jobs, document Q&A pipelines — these workflows repeatedly hammer upstream inference providers (OpenAI, Anthropic, Mistral) with functionally identical prompts.

The result: sky-high monthly inference bills, unpredictable latency spikes, and engineering teams flying blind without fine-grained, per-tenant observability.

This article breaks down how to build a production-grade, drop-in, OpenAI-compatible reverse proxy using async Python (FastAPI), Redis vector search for semantic caching, and OpenTelemetry (OTel) for unified tracing and token metrics.

The Bottleneck: Fragile Commercial Gateways Are Expensive

When engineering leaders realize LLM costs are spiraling out of control, they typically evaluate two paths — both with significant drawbacks.

The first is proprietary observability and gateway platforms. Commercial APM tools and dedicated LLM SaaS gateways either charge recurring per-seat fees or add a premium on top of every million tokens. Worse, sending unredacted tenant prompts to a third-party gateway creates serious data privacy and compliance risks.

The second is naive exact-match caching. Hashing prompts directly (SHA256(prompt)) and storing them in Redis yields embarrassingly low hit rates — often under 4% — because even a single extra space or punctuation mark invalidates the cache.

Traditional Flow:
Client ──> Commercial Gateway ($$$ per token) ──> OpenAI API ($$$ per run)

Optimized Self-Hosted Flow:
Client ──> FastAPI Proxy ──> Redis Vector Search (Cosine >= 0.92) ──> Cache HIT (0ms upstream cost)
                    │ (Cache MISS)
                    └───> Upstream Provider (OTel Metric Exported)

The sustainable solution is to self-host a proxy that serves as an exact drop-in replacement (base_url="http://proxy-host/v1"), validates semantic similarity via vector embeddings, and pushes fine-grained OTel spans directly into Grafana, Jaeger, or Prometheus.

Five-Stage Architecture Pipeline

The entire proxy pipeline executes five sequential stages, each with a clear responsibility.

Intercept & Tenant Auth: Extract Authorization and X-Tenant-ID from request headers, then validate against token budget quotas.

Embed & Similarity Probe: Vectorize the incoming user prompt using a lightweight embedding model (e.g., text-embedding-3-small or a locally deployed SentenceTransformer).

Redis Vector Similarity Query: Run a K-nearest-neighbor (KNN) search over indexed prompt embeddings using cosine similarity. If similarity >= threshold (e.g., 0.92), immediately return the cached completion.

Circuit Breaker & Token Guardrails: On a cache miss, check the tenant's real-time token consumption bucket. If the sliding window budget is exceeded, return HTTP 429 without touching the upstream at all.

OTel Span Finalization: Emit OTLP metrics (prompt tokens, completion tokens, latency, cache status) and export them directly to your APM backend.

The core value of this pipeline: cache hits carry zero upstream cost, while misses are protected by budget guardrails to prevent runaway bills.

Core Code and Implementation Logic

Redis Semantic Cache Index Initialization

First, establish an HNSW vector search index in Redis. We use 1536 dimensions (matching text-embedding-3-small) with cosine similarity as the distance metric.

import numpy as np
from redis.asyncio import Redis
from redis.commands.search.field import VectorField, TextField
from redis.commands.search.indexDefinition import IndexDefinition, IndexType
from redis.commands.search.query import Query

INDEX_NAME = "idx:semantic_cache"
VECTOR_DIM = 1536  # text-embedding-3-small dimension

async def init_redis_indices(redis_client: Redis):
    try:
        await redis_client.ft(INDEX_NAME).info()
    except Exception:
        schema = (
            TextField("prompt_text"),
            TextField("completion_text"),
            VectorField(
                "prompt_vector",
                "HNSW",
                {"TYPE": "FLOAT32", "DIM": VECTOR_DIM, "DISTANCE_METRIC": "COSINE"}
            )
        )
        definition = IndexDefinition(prefix=["cache:prompt:"], index_type=IndexType.HASH)
        await redis_client.ft(INDEX_NAME).create_index(schema, definition=definition)

HNSW (Hierarchical Navigable Small World) is one of the most widely adopted approximate nearest neighbor (ANN) indexing algorithms in production engineering. Its core idea is to build a multi-layer graph structure: upper layers are sparse with long-range links for fast coarse positioning, while lower layers are dense with short-range links for precise search. Queries start from an entry point at the top layer, greedily hopping toward closer neighbors and descending layer by layer until a candidate set is found at the bottom. Compared to brute-force KNN, HNSW achieves single-digit millisecond query latency on million-scale vectors while typically maintaining recall above 95%. The trade-offs are higher memory usage during indexing (each node stores multiple edges) and longer index build times compared to IVF-based algorithms. Redis Stack's built-in redis-search module has natively supported HNSW since version 7.2, with two tunable parameters: M (maximum out-edges per layer, typically 16) and EF_CONSTRUCTION (dynamic candidate set size during build, typically 200). For production deployments, setting EF_RUNTIME (dynamic candidate set size at query time) between 10 and 50 provides a good latency-vs-accuracy operating point.

Semantic Matching Engine

The matching engine performs a KNN search and retrieves the single nearest vector. Pay close attention to the conversion between cosine distance and similarity: a distance of 0 means identical, 2 means completely opposite, so similarity = 1 - distance.

async def check_semantic_cache(redis_client: Redis, query_vector: list[float], threshold: float = 0.92):
    query_bytes = np.array(query_vector, dtype=np.float32).tobytes()
    query = (
        Query("*=>[KNN 1 @prompt_vector $vec AS score]")
        .sort_by("score")
        .return_fields("completion_text", "score")
        .dialect(2)
    )
    results = await redis_client.ft(INDEX_NAME).search(query, query_params={"vec": query_bytes})
    if results.docs:
        doc = results.docs[0]
        distance = float(doc.score)
        similarity = 1.0 - distance
        if similarity >= threshold:
            return doc.completion_text, similarity
    return None, 0.0

Cosine similarity measures the directional angle between two vectors, not Euclidean distance. It's calculated as the dot product of two vectors divided by the product of their magnitudes, yielding a result in the range [-1, 1], where 1 means identical direction, 0 means orthogonal, and -1 means opposite directions. For text embeddings, semantically similar sentences typically have a small angular separation, so their cosine similarity is close to 1. The score field returned by Redis uses cosine distance (1 - cosine similarity), which is why the code must perform the similarity = 1.0 - distance conversion — an easy mistake to miss in implementation. Choosing 0.92 as the default threshold means cache hits only trigger when two texts have very high semantic overlap, corresponding in practice to "different phrasing but identical intent" (e.g., "how do I reset my password" vs. "how do I recover my password"). A threshold that's too low causes semantically unrelated queries to produce false hits (lower precision), while a threshold that's too high approaches exact matching and dramatically reduces the cache hit rate — so threshold tuning is fundamentally an engineering trade-off between precision and recall.

Proxy Route with OpenTelemetry Instrumentation

The proxy route fully replicates the /v1/chat/completions interface. It computes the embedding, checks the cache, and returns immediately on a hit (recording zero tokens). On a miss, it forwards the request to the upstream OpenAI and asynchronously writes the result back to the cache. Key details: every step writes attributes to the span (tenant ID, cache hit status, cosine similarity), and token counters are tagged separately by tenant and hit status.

@app.post("/v1/chat/completions")
async def chat_completions_proxy(request: Request):
    tenant_id = request.headers.get("X-Tenant-ID", "default_tenant")
    payload = await request.json()
    messages = payload.get("messages", [])
    last_prompt = messages[-1]["content"] if messages else ""

    with tracer.start_as_current_span("llm_completion_router") as span:
        span.set_attribute("llm.tenant_id", tenant_id)
        prompt_vec = await compute_embedding(last_prompt)
        cached_response, similarity = await check_semantic_cache(app.state.redis, prompt_vec, threshold=0.92)
        if cached_response:
            span.set_attribute("llm.cache_hit", True)
            span.set_attribute("llm.cosine_similarity", similarity)
            token_counter.add(0, {"tenant": tenant_id, "cache_hit": "true"})
            return {"id": "cached-completion", "object": "chat.completion", "choices": [...], "usage": {"total_tokens": 0}}
        # Cache Miss - Forward to OpenAI
        ...

A cache hit means zero upstream call cost — this is the core mechanism behind the bill reduction.

OpenTelemetry (OTel) is a CNCF-driven observability standard that unifies the collection and export protocols for three signal types: Traces, Metrics, and Logs, designed to replace vendor-locked APM SDKs. Its core concepts include: Span (a record of a single operation, containing start time, end time, attribute key-value pairs, and status), Trace (a directed acyclic graph of causally linked Spans representing a complete request chain), and OTLP (the OpenTelemetry Line Protocol, a standardized data export protocol supporting both gRPC and HTTP transport). In this proxy architecture, each /v1/chat/completions request corresponds to a root Span, with embedding computation, cache lookup, and upstream forwarding each nested as child Spans. By tagging Spans with attributes like llm.tenant_id, llm.cache_hit, and llm.cosine_similarity, downstream tools like Grafana or Jaeger can aggregate cache hit rates and token consumption trends per tenant. This "code as dashboard data source" pattern transforms observability from a post-incident debugging tool into a real-time cost control decision engine.

Deployment and Edge Case Handling in Real-World Scenarios

Pushing a semantic proxy to production requires properly handling three categories of edge cases that will definitely appear in the wild.

Sliding Window Circuit Breaker: If an automated workflow goes haywire (e.g., an n8n webhook caught in an infinite loop), basic rate limiting won't save your bill. Use atomic Redis INCRBY operations keyed by tenant plus the current minute (tenant:{id}:budget:{YYYYMMDDHHmm}) to accumulate consumption. Once the token limit is exceeded, immediately trip the circuit breaker and return HTTP 429.

Dynamic Similarity Thresholds: Not every business can tolerate loose semantic matching. A customer service FAQ bot works well at a threshold of 0.88, while legal or financial extraction tasks may require 0.97 or even deterministic exact matching. Thresholds should therefore be configurable via an HTTP header (X-Semantic-Threshold: 0.95).

Connection Pooling and Downstream Timeouts: Set strict connection timeouts (typically 2.0 seconds) and read timeouts (30.0 seconds) on your HTTPX client pool. Also ensure vector embedding runs asynchronously, keeping the total cache lookup overhead under 12 milliseconds.

These three points may seem like minor details, but they determine whether the proxy becomes an asset or a liability under real traffic pressure.

Conclusion

At this point, you have a complete architectural blueprint: eliminate redundant API calls, enforce per-tenant token guardrails, export enterprise-grade OpenTelemetry metrics — all without paying for software seats.

The core premise behind claiming a 58% bill reduction is that enterprise workloads inherently contain a significant proportion of duplicate queries — semantic caching intercepts that traffic before it ever reaches the expensive upstream. The engineering details of threshold tuning, circuit breaker strategy, and async embedding collectively determine how much you actually save. For teams with heavy LLM inference workloads, a self-hosted proxy like this both controls costs and keeps data sovereignty and observability firmly within your own infrastructure.

Background: Redis INCRBY and Sliding Window Rate Limiting

Redis's INCRBY command is an atomic operation — even with multiple proxy instances writing to the same key concurrently under high load, race conditions that cause undercounting or double-counting do not occur. Combined with the EXPIRE command (setting a key TTL of 60 or 120 seconds), you can build a lightweight fixed-window counter: keyed as tenant:{id}:budget:{YYYYMMDDHHmm}, each request atomically increments token consumption, with an expiration set when the key is first created so it vanishes automatically at the end of the window and starts fresh the next minute. A sliding window requires maintaining multiple time buckets and summing them for smoother rate limiting, but involves slightly more complex logic. Both approaches are far more reliable than in-process memory counters at the application layer, since proxies are typically deployed with multiple replicas — in-memory counters cannot share state across instances, effectively multiplying the rate limit by N (the replica count) and rendering it meaningless.