LiteLLM Router: Say Goodbye to Single-Provider Hardcoding

When OpenAI returns 429 errors for twenty straight minutes, any customer service bot that depends on it grinds to a complete halt. The problem isn't the product itself — it's that the client code has hardcoded a single provider. LiteLLM unifies access to over 100 model providers through an OpenAI-compatible interface, and uses its Router component to retry within the same model group and automatically switch to a backup group when the primary model keeps failing. Business code continues to use the chat completions interface, while reliability management is pushed down into the routing configuration.

Why Single-Provider Hardcoding Breaks Production

Most teams start out writing code like this:

from openai import OpenAI
client = OpenAI()
client.chat.completions.create(model="gpt-4o-mini", messages=...)

This works fine during prototyping, but once you go to production, requirements change: you need a backup provider that can step in during rate limits or outages; message formats need to stay consistent no matter which model responds; retry logic shouldn't have to be rewritten at every call site; and cost and failure logs need centralized management instead of try/except scattered everywhere.

LiteLLM's approach is to keep the OpenAI client usage pattern intact while moving provider selection into a model_list plus Router.

LiteLLM's core abstraction is the OpenAI-compatible layer. OpenAI's chat.completions.create interface has effectively become the industry standard, but the official APIs from Anthropic, Google Vertex, Bedrock, Cohere, Ollama, and others differ in request bodies, response structures, and exception types. LiteLLM does protocol translation in the middle: converting unified inputs into native requests for each provider, then mapping returned results and error codes back to OpenAI structures — for example, translating Anthropic's overloaded_error into RateLimitError. This means business code only needs to understand one data structure, and switching providers doesn't require rewriting parsing logic or maintaining separate exception branches for each SDK.

Installation and a One-Line Call

First, install the SDK:

pip install litellm
# or: uv add litellm

Without a proxy, you can call it directly in SDK style. All providers share the same completion() form:

from litellm import completion
import os

os.environ["OPENAI_API_KEY"] = "..."
os.environ["ANTHROPIC_API_KEY"] = "..."

r = completion(
    model="openai/gpt-4o-mini",
    messages=[{"role": "user", "content": "Summarize this ticket in one line."}],
)
print(r.choices[0].message.content)

r2 = completion(
    model="anthropic/claude-sonnet-4-5",
    messages=[{"role": "user", "content": "Summarize this ticket in one line."}],
)

Every response follows the OpenAI structure. Exceptions are also mapped in OpenAI style, such as RateLimitError and AuthenticationError, so existing error handling code doesn't need to change.

The Key Workflow: Router and Fallbacks

This is the part that actually ships to production. The following four steps form a complete failover loop.

1. Define Deployments with a Shared Alias

import os
from litellm import Router

model_list = [
    {
        "model_name": "support-summarizer",  # app-facing alias
        "litellm_params": {
            "model": "openai/gpt-4o-mini",
            "api_key": os.environ["OPENAI_API_KEY"],
        },
    },
    {
        "model_name": "support-summarizer-backup",
        "litellm_params": {
            "model": "anthropic/claude-haiku-4-5",
            "api_key": os.environ["ANTHROPIC_API_KEY"],
        },
    },
]

2. Configure Retries and Fallbacks

router = Router(
    model_list=model_list,
    num_retries=2,
    fallbacks=[{"support-summarizer": ["support-summarizer-backup"]}],
)

Retries stay within the same model_name group, while fallbacks leave the current group and move to the next group in the list. This separation is intentional: if the primary provider has another available deployment that can respond, you shouldn't waste the backup provider on a single request that failed transiently.

The key here is understanding the distinction between retry and fallback as two orthogonal concepts. num_retries operates within the same model_name group: if support-summarizer has multiple gpt-4o-mini deployments (different keys or regions), a 429 will first trigger in-group retry rotation, because this kind of transient failure often succeeds with a different key. fallbacks, on the other hand, only trigger after the entire group has failed all retries, switching to a different provider — essentially a cross-provider failover. Separating the two matters for both cost and semantics: in-group retries are cheap and have controlled latency, while cross-group fallbacks may introduce output style differences and higher unit costs, so they shouldn't be easily triggered by a single transient failure. Additionally, the Router supports a cooldown mechanism, which temporarily removes persistently failing deployments to avoid repeatedly hitting known-unavailable endpoints.

3. Call It Just Like OpenAI

response = router.completion(
    model="support-summarizer",
    messages=[
        {"role": "system", "content": "One-line summary. No preamble."},
        {"role": "user", "content": ticket_body},
    ],
)
summary = response.choices[0].message.content

The entire ticket pipeline never imports Anthropic. If OpenAI is still rate-limited after retries, LiteLLM automatically switches to the backup alias.

4. Log Requests That Still Fail

Catch litellm.RateLimitError or APIError after fallbacks are exhausted. These inputs can become tomorrow's evaluation set, and they're also a signal to raise your budget or add a new region.

Optional Approach: Use a Proxy Instead of an Embedded Router

If multiple services need to share the same failover rules, you can run the LiteLLM AI Gateway and point all OpenAI clients at it:

from openai import OpenAI

client = OpenAI(
    api_key="sk-your-virtual-key",
    base_url="http://localhost:4000",
)
client.chat.completions.create(
    model="support-summarizer",
    messages=[{"role": "user", "content": "hello"}],
)

A configuration YAML handles model_list, fallbacks, budgets, and virtual keys. The application stays simple; the gateway handles reliability.

The LiteLLM AI Gateway is a deployment form that moves Router capabilities from in-process to a standalone service. It's essentially an HTTP service exposing OpenAI-compatible endpoints like /v1/chat/completions, and applications just need to point base_url at the gateway. Model lists, fallback rules, budget caps, rate limits, and virtual keys are all managed server-side via YAML. The value of virtual keys is this: they're proxy keys issued by the gateway that map to real provider keys but can have independent quotas and scopes, so upstream services never touch the original API Key. Compared to embedding a Router in each service, the proxy model's advantage is centralized rules and changes without redeploying applications; the cost is an extra network hop and single-point operational burden.

When It's Not Worth the Hassle

• If you have just one hobby script, one key, and one model, the native SDK is enough.
• If you already have a mature internal gateway with the same capabilities.
• If your latency budget is extremely tight, an extra hop or cross-provider fallback is worse than a brief 429.

Otherwise, any team that's been burned by a single-provider outage should put fallback configuration right next to the model list, not scattered across a dozen call sites.

Production Checklist

1. Replace scattered provider SDKs with LiteLLM's completion or an OpenAI client pointed at a proxy.
2. Provide app-facing model_name aliases for production tasks, not raw provider IDs.
3. Set num_retries to handle transient in-group failures.
4. Set fallbacks to switch to a different provider or region on sustained failures.
5. Log fallback-exhausted scenarios and treat them as product incidents, not "weird LLM noise."

Reference Resources

• LiteLLM Getting Started Docs
• LiteLLM AI Gateway (Proxy)
• Fallbacks and Provider Failover
• Router: Load Balancing, Retries, Cooldowns
• GitHub: BerriAI/litellm

When your primary LLM provider is rate-limited, are you relying on hope, a second hardcoded client, or an actual router?

Additional Background

On the division of labor between model_name and litellm_params.model, it's worth emphasizing once more: the former is the app-facing alias, and business code depends only on it; the latter is the real routing target, in provider/model-id format (e.g., openai/gpt-4o-mini). The alias mechanism lets the same fallback configuration swap out underlying models wholesale — for example, upgrading support-summarizer from gpt-4o-mini to a new version only requires changing one configuration entry, with no changes needed at any call site. This is also a practical signal for whether you should introduce LiteLLM: if specific model ID strings appear all over your code, then the cost of switching providers is distributed across every call site.