Inside LiteLLM's Architecture: How a Request Travels Through an AI Gateway to 100+ Models

LiteLLM is an open-source AI gateway that sits between your application and 100+ model providers behind a single, unified OpenAI-format interface. Your app talks only to it, and it handles auth, routing, translation, billing, and accounting. This article follows a real Hello! request and breaks down the design and trade-offs of each stage along the way.

What LiteLLM is: the company mailroom

LiteLLM's positioning can be summed up in one sentence: it's an open-source gateway that every AI request passes through before being forwarded to the actual model provider, such as OpenAI, Anthropic, Gemini, Bedrock, and Azure.

It ships in two forms: a Python SDK you call directly from your own code, and a gateway (also called a proxy) deployed and shared by a team or company. According to ARCHITECTURE.md in the project repo, the gateway uses the SDK internally and layers identity verification, usage limits, budgets, and routing on top. This article focuses on the gateway form.

A good analogy is the company mailroom: the clerk verifies the sender, picks a courier, repackages the parcel to the courier's requirements, and finally records the postage. The business teams upstairs never need to learn each courier's rules.

The full journey of a request

Suppose an application sends Hello! to gpt-4o through the gateway. The whole process can be viewed along four lines: what comes in, who does the work, what gets left behind, and what goes out.

What comes in

The application calls the gateway the same way it would call OpenAI, e.g. hitting /chat/completions on port 4000. The Authorization header carries a virtual key — a key the gateway generated for a team or application, with the real provider key hidden in the backend.

Two other inputs matter just as much: the public price table loaded at startup, and the config.yaml configuration file. The latter maps each model name to a model group, and a group can contain multiple deployments, each representing a specific place where the model runs — another key, another region, or another provider.

Who does the work

The first step is fetching the key and budget. They're read from in-memory or Redis cache, and the database is only queried on a cache miss. Next comes rate limiting: separate caps on request count and token count, enforced at four levels — server, key, user, and team.

The router then picks a deployment from the model group, defaulting to simple random spread. Finally, the provider-specific translation file converts the OpenAI format into the provider's format, and converts the response back to OpenAI format.

Virtual keys and deployments

A virtual key is a key the gateway issues itself. Its format is usually still sk-..., but its value is stored in Postgres and associated with a team, user, or budget object. The application uses this key to call the gateway, and only after looking it up does the gateway know which real provider key to use for the upstream call. The real key never appears on the client. This brings two immediate benefits: rotating a provider key doesn't require redistributing client config, and you can revoke a single virtual key without affecting other callers.

A deployment is one concrete physical location of the same model: another key from the same provider, another regional endpoint, or an equivalent model from a different provider altogether. A model group is a logical name (like gpt-4o), and the deployments inside it are interchangeable instances. The router only ever chooses among deployments, so "the same model name" is really a set of candidate backends inside the gateway.

After the answer: cost, logs, and records

Before the response goes out, the gateway does the math: input and output token counts times the unit prices in the price table. Per ARCHITECTURE.md, this cost is returned to the application in a response header called x-litellm-response-cost.

What goes out, what gets left behind

The application receives a response in OpenAI format. Logs are sent in the background to observability tools like Langfuse and Datadog, while the admin UI shows keys, teams, and spend.

For persistence, Postgres stores keys, teams, budgets, and spend logs, with the schema defined in Prisma. Redis is optional; when present, it holds caches, rate-limit counters, and the spend buffer.

The spend from a single Hello! first goes into a queue and is written to Postgres in batches by a background job, by default every 10 to 15 seconds. That means there are no database writes anywhere on the request path.

The spend buffer

This queue isn't a message broker — it's a batch of pending records maintained inside the gateway process (or in Redis). When a request finishes, the gateway simply appends that call's token counts, cost, and key ID to the buffer and returns the response immediately; an async background task flushes the whole batch to Postgres at a fixed interval. Batch writes turn N requests into a single database transaction, significantly reducing connection overhead and write amplification.

The cost is visibility lag: a request you just sent won't show up in the admin spend report until the next refresh cycle. Reconciliation workflows need to account for this, e.g. by querying with a time window rather than expecting immediate visibility.

Four key trade-offs and what each costs

LiteLLM's architecture is no free lunch — every design choice has a clear price.

Unified OpenAI format. No adapter layer per SDK. The benefit is that switching providers means changing a model name, not code; the cost is that capabilities a provider has but the OpenAI format lacks must be covered by extra parameters.

Keeping the database off the request path. The gateway sits in front of every call and must never add latency; the cost is that spend totals in the database lag, budget checks rely on shared real-time counters, and multiple replicas can only synchronize through Redis.

Prices as a public file. Model prices live in a JSON file in the repo, with a copy bundled in the install package. Files were chosen over a pricing service or provider billing APIs because anyone can read, modify, and reuse them. The cost is like a menu printed in the morning: if the kitchen raises prices at noon, the menu on the table still shows the old ones until someone reprints it. A running proxy keeps using the prices loaded at startup unless an admin manually reloads them or sets up a scheduled job, so costs can drift.

Redis only needed for horizontal scaling. A single replica keeps caches and counters in memory with no extra components; the cost is that without Redis, limits and caches are tracked per replica.

What happens when you scale tenfold

Imagine ten gateway replicas behind a load balancer. Rate limiting and caching move to Redis, so replicas agree on usage per key; spend updates can also land in Redis first and be written in batches. The Terraform modules in the repo go further, splitting the gateway and admin UI into two separately deployed services.

So what happens when a provider goes down? Think of it like a phone call: if one clerk is busy, you're transferred to another clerk at the same store; only when the whole store closes do you call the next store. In LiteLLM, a retry asks the same model group again and may land on another deployment; a fallback switches to the next model group in config when the first keeps failing.

Retry vs. fallback

A retry happens within the same model group: after a failed call (timeout, 5xx, rate limit), the gateway retries the configured number of times, and the router's choice may land on another deployment in the group — so a retry inherently means "try a different backend." A fallback crosses model groups: when the first group exhausts its retries and still fails, the gateway switches to the next model group in the config.yaml fallback list, potentially changing provider and model.

Both add latency and carry a risk of duplicate billing, so in practice retries are usually set to a small count, and fallback is used cautiously for write operations that are sensitive to idempotency.

Verify it yourself

Follow the README and you can see this machinery on your own machine.

Install the gateway with uv tool install 'litellm[proxy]', put your OpenAI key in OPENAI_API_KEY, then start it with litellm --model gpt-4o. Point any OpenAI client at port 4000 on localhost (the README uses 0.0.0.0:4000) and send Hello!; with curl, request /v1/chat/completions and print the response headers.

Find the x-litellm-response-cost header — that's the cost the gateway computed for this call. Then open model_prices_and_context_window.json in the repo, find your model, and multiply the unit prices by the input and output token counts in the response's usage. If the two numbers don't match, first check whether the price file was modified after the gateway loaded it.

One operational detail worth noting

From an engineering standpoint, LiteLLM treats "don't block the request" as the top priority, and this choice runs through all three paths: auth, billing, and rate limiting. The price it pays is equally concrete: a 10-to-15-second lag window on spend data, a price table that can go stale, and Redis becoming mandatory for multi-replica deployments.

For single-machine or small-team setups, these costs are nearly negligible; but when logs need to reconcile with finance, or when budget control demands strong consistency, price drift and spend lag become areas that need extra tooling as a safety net. That's exactly why the original author built a browser-based reconciliation tool that checks gateway spend logs against daily prices.