LLM Proxy Benchmark: 200 Identical Requests Collapse into 1 Call, but a 30-Second Performance Bug Surfaces

An open-source Rust LLM proxy demonstrated impressive deduplication in benchmarking: 200 identical requests merged into a single upstream call. But the same test also exposed a serious bug the author hadn't noticed—serial server processing caused 20 distinct requests to take 30 seconds instead of the expected 1.5 seconds.

What an LLM proxy is and what problem it solves

An LLM proxy is a small server sitting between your application and the model API. Clients don't need code changes—just point the base URL elsewhere—and the proxy can coalesce identical requests, circuit-break failing providers, cap spending, and log every call.

The open-source project, called tokio-prompt-orchestrator, is MIT-licensed and developed by Mattbusel. In October 2026 he ran a round of real-world benchmarks, placing the official OpenAI SDK between the proxy and a mock provider to measure actual proxy behavior.

The setup was clever: same SDK, same application code, with the only change being whether the base URL pointed directly at the provider or at the proxy. The provider was a 60-line Node server that speaks both OpenAI and llama.cpp formats, supports both 503 and hang failure modes, and precisely counts every call that arrives—exactly the number the provider bills you for.

Test 1: 200 identical requests become 1 paid call

This is the proxy's flagship scenario: the same prompt fired 200 times simultaneously. In the real world this could be a cache miss on a hot page, duplicate message-queue consumption, a user mashing the button, or an agent framework retrying a slow call.

Direct: 200 requests produce 200 upstream calls.

Through the proxy: just 1 upstream call. Response headers clearly recorded each request's fate—1 miss (initiated the call), 199 joined (waited and shared the result). The slowest response was still 1.58 seconds; coalescing added no extra latency.

There are two details in the code worth borrowing. First, the dedup key is a hash of the entire flattened conversation, not just the last message, so users typing the same text in different chats don't share answers. Second, the upstream call runs in a separate spawned task—if the first caller hangs up, the 199 joined requests still get their result. Without this design, one impatient client would cancel everyone's call. Go programmers will recognize this as the singleflight pattern.

Completed answers are reusable within the default 300-second window, stored in a size-limited moka cache to prevent unbounded memory growth.

singleflight pattern: The article notes Go programmers will recognize this as the singleflight pattern—a concurrency primitive that coalesces multiple concurrent requests for the same key at the same moment into a single actual execution, with the remaining callers waiting and sharing the result. Its core value is eliminating "thundering herd" duplicate work: when a cache misses or a downstream service is slow, a flood of identical requests rushes toward an expensive resource at once. Implementations typically maintain a mapping from key to "in-flight call"; the first caller creates and executes the task, later callers attach to it and wait; the mapping is deleted immediately once the task completes, avoiding stale cached results. Unlike ordinary caching, singleflight only coalesces in-flight requests, doesn't change request semantics, and has near-zero overhead on a miss. The proxy in this article uses the full conversation hash as the key and runs the upstream call in a separate task, ensuring that the remaining requests still get their result if the first caller disconnects.

In real traffic, dedup saves only 1.5%

But load testing is the best case. The author replayed the Allen Institute for AI's public WildChat-1M dataset—conversation logs between real users and the free ChatGPT frontend. He took the first parquet shard: 59,857 conversations, 172,418 user turns, 25.3 days of traffic.

With the default 5-minute window, only 2,568 of 172,418 calls (1.49%) were exact duplicates. Stretching the window to 24 hours brings it to just 4.39%, and the full 25 days only 4.77%.

The breakdown is more interesting: within 5 minutes, 91% of duplicates came from the same user (same hashed IP) resending identical content—refreshing, resubmitting, "it didn't answer, let me try again." The most-duplicated request wasn't even from a human: the system prompt used by AutoGPT in 2023 (starting with "CONSTRAINTS: 1. ~4000 word limit...") repeated 76 times within the 5-minute window, along with scripted fill-in-the-blank test prompts. Duplication is truly concentrated in agent loops and scripted requests.

So the rule is: dedup should be on by default (zero cost on a miss), but real savings only appear when machines send the requests. Handling paraphrase is semantic dedup—a harder problem.

Test 2: circuit breaker blocks 94% of futile calls during provider outage

A different prompt every 2 seconds (dedup can't help), sustained for 150 seconds. The provider was healthy for the first 15 seconds, returned 503 from 15 to 75 seconds, then recovered.

Direct: 89 calls fired at a dead provider within 60 seconds. That's 30 requests each retried 3 times—the SDK automatically retries twice on every 5xx. Your application tripled the load on an already-wobbling provider. This is exactly the AWS Builders' Library warning about retries: retries help with small glitches but pour gasoline on a real outage.

Through the proxy: just 5 calls. After 5 consecutive failures the circuit breaker opened, and subsequent requests were rejected inside the proxy without touching the provider at all. A 94% reduction in calls to the failing API, with zero configuration.

Circuit breaker is the classic pattern Martin Fowler described in 2014: closed (calls pass), open (calls rejected), half-open (probe for recovery). Defaults: triggers on 5 consecutive failures, closes at 80% recent success rate, half-opens after 60 seconds.

But two honest caveats. First, users wait just as long for the error: about 1.3 seconds either way. The proxy rejects immediately, but the SDK sees the 503 and still retries twice with 0.5s and 1s backoff. The proxy didn't send x-should-retry: false or a Retry-After header on breaker rejections—and the OpenAI SDK actually honors that header. It's a one-line fix.

Fixed in 2.0.3: breaker rejections now carry x-should-retry: false and Retry-After, returning rejections in about 1 millisecond instead of 1.35 seconds, and p95 dropped from 1459 ms to 314 ms.

Second, recovery lags. The breaker opened at 18.6 seconds but didn't look again until 60 seconds after the last failure, so traffic only recovered at 78.8 seconds. The provider was healthy again at 75 seconds, but the request at 76 seconds still got a 503. It's a trade-off: fewer calls to a sick API, at the cost of slower return to health.

Circuit breaker pattern: The article mentions the circuit breaker several times; here are its state machine and design points. A circuit breaker typically has three states: closed, where requests pass normally while failure rates are tracked; when consecutive failures or the failure ratio exceed a threshold, it moves to open, where all requests are rejected immediately without touching downstream; after a cooldown it enters half-open, allowing a small number of probe requests—success returns it to closed, failure reopens it. Unlike plain retries, a circuit breaker aims to fail fast, protect an already-overloaded downstream, and give it time to recover. Key parameters include the trigger threshold, cooldown duration, and number of half-open probes. The article's defaults—trigger on 5 consecutive failures, half-open after 60 seconds—explain the recovery lag: during cooldown, requests are rejected even if the provider has recovered, trading availability for protection against a sick dependency.

Test 3: a hanging provider tests your timeout settings

The sneakier failure is a provider that accepts connections but never responds. Same load, but this time the provider hangs.

Neither side failed fast. Direct p95 wait was 53.7 seconds; the first request during the outage waited 59.7 seconds until the provider dropped the stuck connection at 75 seconds and the SDK's retry got through. If the provider never dropped it, the SDK's default 10-minute timeout would leave every request hanging for 10 minutes, retried 3 times.

Through the proxy p95 was 53.6 seconds, essentially the same. Only 2 calls reached the hanging provider (30 direct), but the breaker never opened—the llama.cpp worker times out at 30 seconds, and the breaker needs 5 failures, i.e. 150 seconds of hanging, to trigger, but the outage ended before that.

2.0.3 added a --timeout flag and counts timeouts as breaker failures. Previously the pipeline's timeout wrapped the breaker call, so hanging calls cancelled by the pipeline never counted as failures. With --timeout 5, the breaker opened about 13 seconds into the hang, the slowest wait dropped from 66.3s to 11.4s, and median wait during the hang fell from 34.9s to 1 ms. The cost is that those requests fail fast with a 503 instead of succeeding a minute later—which is exactly the point of a circuit breaker.

The most practical lesson: the breaker can't be faster than your timeout. Five failures at a 30-second timeout means 2.5 minutes of hanging before it acts. If your model usually responds in 5 seconds, set the timeout near your actual p99 (say 15–20 seconds), not the default 30 seconds, 120 seconds, or 10 minutes.

p95 latency and timeout settings: The article repeatedly uses p95 to measure wait times—the value below which 95% of requests complete—a tail-latency metric. Compared with averages, p95 better exposes how a few very slow requests affect user experience. Timeouts should match the actual latency distribution: if the model's normal p99 is 15–20 seconds, a timeout around 20 seconds fails fast during hangs without killing legitimately slow requests. SDK defaults are often as long as 10 minutes, unacceptable for interactive applications. The article stresses that "the breaker can't be faster than your timeout," because the breaker relies on failure counts, and a hanging request only counts as a failure after it times out; the longer the timeout, the longer the total hang needed to reach the trigger threshold. Counting timeouts as breaker failures and shortening timeouts is what lets the breaker open promptly when a provider hangs.

The bug found: one upstream call at a time

To check concurrency, the author sent 20 distinct prompts simultaneously, with the provider taking 1.5 seconds each. Dedup can't help here; this is pure throughput.

Direct: 1.5 seconds per answer. Through the proxy: a staircase—1.5s, 3.0s, 4.5s... up to 30.2 seconds. The proxy made the same 20 calls, but strictly serially. The server binary serialized every upstream call.

Fixed in 2.0.3: the server now runs up to 16 upstream calls concurrently (--concurrency), and the same 20 distinct prompts complete in 3.0 seconds at default settings and 1.5 seconds with --concurrency 32—same as direct. Identical prompts still share one call: the 200-request crowd is still 1 upstream call.

The cause was in src/stages.rs. The inference stage was a single Tokio task that took a request from the channel and awaited the model before taking the next. The library actually already had the fix—spawn_pipeline_with_config, which runs a pool of workers sharing the channel, sized by inference_workers. But it defaulted to 1 for backward compatibility, and the orchestrator server binary used the simpler spawn_pipeline_with, always a single worker. The README demo (12 requests, 3 model calls) never caught this because it was built to showcase dedup—and dedup conveniently masked the problem.

This is the most painful and most valuable part of benchmarking your own code: the dedup and breaker numbers matched expectations, but the staircase chart didn't—and that's a problem any stranger would hit in their first hour.

Tokio tasks and worker pools: The article notes the inference stage was a Tokio task that took a request from the channel and awaited the model before taking the next, causing serialization. Tokio is Rust's async runtime; a task is a lightweight unit of concurrency scheduled by the runtime, and await means the current task yields execution while waiting for I/O—but if a single task sequentially awaits each upstream call, overall throughput is still limited by that task's serial loop. To process multiple requests concurrently you need multiple tasks sharing one channel, i.e. a worker pool: each worker independently takes requests from the channel and awaits, advancing multiple upstream calls simultaneously. The spawn_pipeline_with_config mentioned in the article does exactly this—starting a worker pool sized by inference_workers—while the server binary mistakenly used the single-worker simplified entry point, causing the staircase serialization of 20 distinct prompts. After the fix, the default is 16 workers, restoring throughput comparable to direct connections.

Actionable advice for your project

Most improvements don't require the author's crate. Any application calling an LLM API should do these things, in order of payoff:

1. Set a realistic timeout. The SDK default is 10 minutes. Measure the p99 latency of the model you use and set slightly above it. In Node: new OpenAI({ timeout: 20_000 }); in Python: OpenAI(timeout=20.0). This one setting decides whether an outage makes users wait 20 seconds or 10 minutes.

2. Understand how retries behave during an outage. 2 retries means 3x load on a struggling provider. Keep retries for small glitches, but put something in front that detects a real outage and stops calling (a circuit breaker)—and have it say "don't retry" when rejecting.

3. Coalesce identical in-flight calls. If anything in your system can send the same request simultaneously (queues, proxies, retries, cron jobs, cache misses), a singleflight map keyed on the full conversation hash turns N calls into 1. Thirty lines in most languages, zero cost on a miss.

4. Don't expect caching to halve your chat bill. Exact duplicates in real chat traffic are only 1.5%. Spend effort on provider-side prompt caching, smaller models for simple calls, and shorter prompts.

5. Load-test your proxy with distinct prompts, not identical ones. Identical-prompt tests flatter any proxy with dedup. The author's own demo did exactly that and hid a 20x slowdown.

If you want a ready-made solution: cargo install --locked tokio-prompt-orchestrator --features web-api,tantivy (gets the 2.0.3 fixes), then point at it with OPENAI_BASE_URL=http://127.0.0.1:8080/v1. It supports the OpenAI and Anthropic APIs, and the official SDKs work unchanged.