Rust-Built World's Fastest AI Gateway: Antirion Benchmarks and Architecture Deep Dive

Why AI Gateway Speed Matters So Much

When LLM features move from prototype to production, the problems usually start with an OpenAI key in a .env file: the team wants to add Claude, OpenAI goes down on a Friday afternoon with no fallback, finance asks how much each team is spending, and legal worries about customer data leaking into prompts. AI gateways exist precisely for this—they sit between applications and model providers, handling routing, fallback, rate limiting, budgets, guardrails, and logging.

The problem is that the gateway sits on the critical path of every single call. If it adds 20ms of latency, then every request the company makes will forever carry that extra 20ms. The author of the open-source gateway Antirion built one from scratch in Rust based on exactly this premise, claiming the world's fastest AI gateway, and benchmarked it against the Go-written Bifrost on the same 4 vCPU machine.

Benchmarks: The Performance Gap at 5,000 req/s

The author tested both gateways on a simple chat scenario at 5,000 req/s:

Metric	Antirion	Bifrost 2.2.4	Gap
Overhead p50	0.53 ms	19.7 ms	37× lower
Overhead p99	0.95 ms	95.2 ms	100× lower
CPU per request	247 µs	717 µs	2.9× less
Peak memory	215 MiB	1,389 MiB	6.5× less
Idle memory	41 MiB	498 MiB	12× less
Max sustained throughput	14,000 req/s	3,500 req/s	4× higher

In the same test, LiteLLM failed every single request at that rate. More notable is the behavior as load increases: at 10,000 req/s, Bifrost still responded but requests began queuing, with median request wait times reaching 12 seconds; Antirion's p99 stayed under 2ms. At 64 concurrent connections, Antirion reached 21,512 req/s versus Bifrost's 6,568 req/s.

The author is candid about fairness: under light load of 1,000 req/s, the two are close—Bifrost p50 at 0.72ms and Antirion at 0.76ms, a tie within noise. The gap only emerges as load grows, which is exactly when a gateway actually matters.

Nine Design Decisions That Make the Performance Possible

1. One Thread per CPU Instead of Work Stealing

Tokio's default multi-threaded runtime migrates tasks between threads, causing cache invalidation and cross-thread synchronization. Antirion instead uses one data thread per CPU, each running a single-threaded runtime: on Linux, each thread listens independently through its own SO_REUSEPORT group, with the kernel handling connection distribution; a client connection, the tasks it spawns, and the HTTP/1.1 connection to the provider all stay on the same thread. Outbound connection pools are partitioned per thread, and a request never wakes another thread from start to finish.

The management layer (console, Admin API, background tasks) runs on a separate multi-threaded runtime and doesn't compete for the hot path. HTTP/2 is the weak point of this model—a single connection can carry hundreds of concurrent requests, so requests on an HTTP/2 connection are distributed round-robin across data threads.

Work stealing is the default scheduling strategy for multi-threaded async runtimes like Tokio: each worker thread maintains its own task queue, and idle threads "steal" tasks from the tail of other threads' queues. This strategy achieves decent load balancing, but the cost is that tasks may migrate across CPU cores, causing L1/L2 cache invalidation, and cross-thread task transfers require atomic operations and synchronization primitives. For scenarios like gateways that do little computation per request but are highly sensitive to tail latency, cache invalidation and synchronization overhead get amplified. Antirion's one-thread-per-core model sacrifices some dynamic balancing capability in exchange for the determinism of binding a request to the same core throughout and eliminating cross-thread wakeups—this is precisely one of the keys to pushing p99 overhead down to sub-millisecond levels.

SO_REUSEPORT is a socket option introduced in Linux 3.9 that allows multiple sockets to bind to the same IP and port, with the kernel distributing new connections to one of the listening sockets via four-tuple hashing. Combined with "one listening socket per CPU, each handled by an independent thread," this achieves kernel-level connection sharding, avoiding the traditional accept thundering herd and single-point distribution bottlenecks.

2. Never Copy the Request Body

Long-context prompts easily reach hundreds of KB, while a gateway typically only needs to modify one or two fields (such as rewriting model). The author wrote a small JSON parser that does strict validation (depth limits, duplicate keys, error messages with line and column) and member position recording in a single pass. On egress, the request body is composed of slices of the client's original bytes, each sent as a separate frame; when the provider and client dialects match, the request body is never copied.

Zero-copy in network proxying means avoiding repeatedly copying payloads between userspace buffers. A typical gateway reads the full request body into memory, deserializes it into a structured object, modifies fields, and serializes it back out—under large prompts this produces multiple full copies and extra memory usage. Antirion's approach is to use a custom parser only to record the offsets of each JSON member in the original bytes, then write out the original bytes directly as separate frames sliced by fragment; only when a field needs rewriting is the corresponding fragment replaced. This way the request body is never copied wholesale when dialects match, reducing both memory bandwidth pressure and latency jitter from large object allocations.

3. Dialect Translation That Doesn't Attempt the Impossible

OpenAI SDK clients can call Claude, and vice versa. The middle layer is an intermediate representation: each client dialect has a decoder, and each provider has an encoder. The most valuable trick: the encoding does a dry run before routing, caching the result per deployment. If a deployment can't serve the request (say the target lacks an equivalent feature), it's eliminated before the first attempt, avoiding fallback wasting a round trip only to discover the inevitable failure.

Dialect here refers to the request/response format differences between model provider APIs—for example, OpenAI's /v1/chat/completions and Anthropic's /v1/messages differ in message structure, system prompt placement, tool-call expression, and streaming event types. The gateway decouples through an intermediate representation (IR): the client dialect is decoded to IR, then the target provider's encoder converts it to that provider's format.

Dry run pre-check means running the encoder once before actually making the upstream call to determine whether a deployment can fully express the request (e.g., the target doesn't support a certain type of multimodal input or parameter). If not, that deployment is eliminated at the routing stage rather than waiting for the fallback to return an error and retry. This avoids wasting a network round trip on a "doomed" request, which is especially critical for tail latency when there's a multi-provider fallback chain.

4. Streaming with Commit Point and Byte-Bounded Queues

Before the first byte reaches the client, the gateway can still switch providers if the upstream fails or a guardrail blocks the request; after that, it can't. So the response is held until the commit point. After that, a dedicated task relays events through a queue bounded by bytes (not by messages), so slow clients apply backpressure upstream rather than accumulating memory in the gateway.

5. Immutable Config with Generational Swapping

The entire config (YAML, Admin API, or console) is validated against a JSON Schema, and each change generates a new generation: the compiled snapshot is built on a blocking thread, persisted, and then swapped in atomically. The snapshot lives in ArcSwap, and each request loads it once at the start and holds it until the end—config changes never take effect mid-request, and the hot path never contends for locks with config writers.

6. Precise Billing Exactly Once per Request

Rate limits and budgets are reserved at admission based on token estimates, then settled precisely once at request end using actual usage, with one ledger line per upstream call. A key detail: each request runs on a tracked task and isn't tied to the connection. If the client disconnects mid-request, the request isn't cancelled, and the cost of what the provider already generated is still counted; if the task panics, the reservation is refunded.

7. Canonical-Key Caching and Single-Flight

The exact cache is keyed on the canonical form of the request body (JSON canonicalization per RFC 8785), so key order or whitespace never causes a miss. When multiple identical requests miss simultaneously, only one goes to the provider and the rest wait for its result.

8. Architecture Enforced by the Build System

The code is divided into 8 layers across 28 crates, and each crate can only depend on layers below it. cargo xtask layers checks run in CI, and violating the rule fails the build.

9. Measurement Methodology That Doesn't Lie to Itself

The author points out that the most common mistake in proxy benchmarking is closed-loop load: the generator waits for each response before sending the next, so when the proxy stalls the generator slows down too, and queuing never shows up in latency (coordinated omission). The test framework uses open-loop scheduling, with latency measured from the scheduled send time; the generator and mock upstream run on separate machines; overhead is computed as the difference between going through the gateway and hitting the mock directly at the same rate; the three gateways run alternately each round, taking the median of five; and validity rules discard disqualified repetitions.

An unexpected finding: the self-test requires the generator-to-mock p99 to be 50µs, but this never passed on AWS—bare TCP connections between two machines in the same placement group already hit p99 of 0.5 to 0.8ms. Since overhead is computed by subtracting each repetition's own direct-connection result, this doesn't affect the comparison, but it shows how much the network influences sub-millisecond numbers.

Coordinated omission, introduced by Gil Tene, refers to the phenomenon in closed-loop load generators where the generator slows its send rate in sync when the system under test slows down, so requests the system should have received are never sent, and queuing latency is systematically underestimated in the statistics. Open-loop scheduling sends requests on a fixed schedule regardless of whether responses return, with latency measured from the "scheduled send time," so queuing time is faithfully counted. This is the most commonly overlooked methodological trap when evaluating tail latency of middleware like gateways and databases.

Open loop vs. closed loop can be summarized as: closed loop is "send the next after receiving a response," close to some real user behavior but masking overload; open loop is "send at a fixed rate," exposing the system's queuing and rejection behavior under pressure. Production traffic is closer to open loop, so gateway benchmarks should be primarily open-loop, with closed-loop only as a supplement.

Two-Minute Quickstart and Current Status

All you need is Docker and an OpenAI or Anthropic key—no config file required:

docker run -p 8080:8080 -p 8081:8081 \
  -v antirion-data:/var/lib/antirion eliasbiondo/antirion

First startup prints the console URL and admin token. Open http://localhost:8081/ui/, add a provider, create a Model named chat, create an API key, then point any OpenAI SDK at http://localhost:8080/v1. Request logs show the routing, tokens, latency, and cost of each request.

Antirion is open source under Apache-2.0, currently at v0.1.1, still rough around the edges. The author raises several open questions at the end: What gateway do you use now, and what bothers you most about it? Should streaming responses be held to the commit point, or should the first byte go out as early as possible and give up fallback? What in the benchmarking methodology can be questioned? The answers to these questions may shape a gateway project's direction more than performance numbers do.

One-line Summary

Rust gateway Antirion with per-core threading and zero-copy design has overhead dozens of times lower than the Go version under high load.

Paragraph Summary

The article introduces Antirion, an open-source AI gateway built from scratch in Rust, arguing that the gateway sits on the critical path of every call and its latency is permanently added to all company requests. In a comparative test on a 4 vCPU machine at 5,000 req/s, Antirion's overhead was p50 0.53ms and p99 0.95ms, far below the Go-written Bifrost; peak memory 6.5× lower and max throughput 4× higher, while LiteLLM failed entirely at that rate. The author details nine design decisions: one thread per CPU with SO_REUSEPORT replacing work stealing, no request body copying, dry run before dialect translation, streaming with commit point and byte-bounded queues, immutable config swapped by generation, precise billing, canonical-key caching with single-flight, build-enforced layering, and an open-loop benchmarking methodology with separate machines and direct-connection subtraction, while candidly admitting the two are close under light load.