tokio-prompt-orchestrator
An open-source Rust LLM proxy server that sits between applications and model APIs. It supports request deduplication in singleflight mode, circuit breaking, rate limiting and logging. Clients only need to change their base URL to use it.
Timeline (last 90 days)
The open-source project tokio-prompt-orchestrator is an LLM proxy server that sits between the application and the model API, is under the MIT license, and is developed by Mattbusel.
In a benchmark of 200 identical requests, connecting directly to the provider produced 200 upstream calls, whereas going through the proxy produced only 1 upstream call, with 199 requests joining the shared result, and the slowest response was still 1.58 seconds.
The proxy's deduplication key is a hash of the entire flattened conversation rather than just the last message, to avoid users who type the same text in different chats sharing an answer.
The proxy's upstream call runs in a separate spawned task, so even if the first caller disconnects, the remaining joined requests can still obtain the result.
Under the default 5-minute window on the WildChat-1M dataset (59,857 conversations, 172,418 user turns, 25.3 days of traffic), only 2,568 calls (1.49%) were exact duplicates; with the window stretched to 24 hours it was 4.39%, and across the full 25 days it was 4.77%.
In the provider-outage test, the direct-connection approach sent 89 calls to the dead provider within 60 seconds (30 requests each retried 3 times), whereas going through the proxy sent only 5 calls, a 94% reduction, because the circuit breaker opened after 5 consecutive failures.
In the hanging-provider test, the p95 wait times for direct connection and the proxy were 53.7 seconds and 53.6 seconds respectively, with only 2 calls from the proxy reaching the hanging provider (versus 30 for direct connection), but the circuit breaker never opened, because with a 30-second timeout it takes 5 failures, i.e. 150 seconds of hanging, to trigger.
Version 2.0.3 changed the circuit-breaker rejection response to carry the `x-should-retry: false` and `Retry-After` headers, so that rejected requests return in about 1 millisecond instead of 1.35 seconds, and p95 dropped from 1459 milliseconds to 314 milliseconds.
Version 2.0.3 added the `--timeout` parameter and counts timeouts as circuit-breaker failures; with `--timeout 5`, the circuit breaker opened after about 13 seconds of hanging, the slowest wait dropped from 66.3 seconds to 11.4 seconds, and the median wait during hanging dropped from 34.9 seconds to 1 millisecond.
Version 2.0.3 fixed the bug where the server processed upstream calls serially and added the `--concurrency` parameter, which by default runs up to 16 upstream calls simultaneously; the same 20 distinct prompts completed in 3.0 seconds under default settings and in 1.5 seconds with `--concurrency 32`.
3 more timeline events
All Facts (13)
Completed answers can be reused within the default 300-second window and are stored in a size-limited moka cache to prevent unbounded memory growth.
50%UnverifiedIn a benchmark of 200 identical requests, connecting directly to the provider produced 200 upstream calls, whereas going through the proxy produced only 1 upstream call, with 199 requests joining the shared result, and the slowest response was still 1.58 seconds.
50%UnverifiedThe proxy's deduplication key is a hash of the entire flattened conversation rather than just the last message, to avoid users who type the same text in different chats sharing an answer.
50%UnverifiedThe proxy's upstream call runs in a separate spawned task, so even if the first caller disconnects, the remaining joined requests can still obtain the result.
50%UnverifiedUnder the default 5-minute window on the WildChat-1M dataset (59,857 conversations, 172,418 user turns, 25.3 days of traffic), only 2,568 calls (1.49%) were exact duplicates; with the window stretched to 24 hours it was 4.39%, and across the full 25 days it was 4.77%.
50%UnverifiedIn the provider-outage test, the direct-connection approach sent 89 calls to the dead provider within 60 seconds (30 requests each retried 3 times), whereas going through the proxy sent only 5 calls, a 94% reduction, because the circuit breaker opened after 5 consecutive failures.
50%UnverifiedIn the hanging-provider test, the p95 wait times for direct connection and the proxy were 53.7 seconds and 53.6 seconds respectively, with only 2 calls from the proxy reaching the hanging provider (versus 30 for direct connection), but the circuit breaker never opened, because with a 30-second timeout it takes 5 failures, i.e. 150 seconds of hanging, to trigger.
50%UnverifiedThe circuit breaker's default configuration is to open after 5 consecutive failures, close at an 80% recent success rate, and half-open for probing after 60 seconds.
50%UnverifiedDeduplication should be enabled by default because it costs nothing on a miss; but real savings only appear in scenarios where machines send requests (agent loops, scripted requests), and semantic deduplication that handles paraphrasing is a harder problem.
50%UnverifiedThe open-source project tokio-prompt-orchestrator is an LLM proxy server that sits between the application and the model API, is under the MIT license, and is developed by Mattbusel.
50%UnverifiedVersion 2.0.3 changed the circuit-breaker rejection response to carry the `x-should-retry: false` and `Retry-After` headers, so that rejected requests return in about 1 millisecond instead of 1.35 seconds, and p95 dropped from 1459 milliseconds to 314 milliseconds.
50%UnverifiedVersion 2.0.3 added the `--timeout` parameter and counts timeouts as circuit-breaker failures; with `--timeout 5`, the circuit breaker opened after about 13 seconds of hanging, the slowest wait dropped from 66.3 seconds to 11.4 seconds, and the median wait during hanging dropped from 34.9 seconds to 1 millisecond.
50%UnverifiedVersion 2.0.3 fixed the bug where the server processed upstream calls serially and added the `--concurrency` parameter, which by default runs up to 16 upstream calls simultaneously; the same 20 distinct prompts completed in 3.0 seconds under default settings and in 1.5 seconds with `--concurrency 32`.
50%