Telnyx AI Gateway Delivers Observable Model Failover

When your application depends on AI models, a critical question is often overlooked: when the primary model fails and you fall back to a backup model, do you really know what happened? Which model actually handled the request? How many attempts did the gateway make? Client-side retry loops can obscure these answers, and SDK retries stacked on top of gateway retries inflate request counts and distort attempt counts.

Telnyx AI Gateway offers a solution that centralizes retry and fallback policies at the gateway layer, making the entire process observable through response headers and a consumption ledger. The recently open-sourced gateway-fallbacks Python example fully demonstrates how to configure primary and backup models, force timeouts to trigger failover, and verify results in two ways.

Architecture: Separation of Control Plane and Inference Plane

The example divides the system into a control plane, an inference plane, and an observation plane. The control plane uses a Telnyx account API key to manage token groups, token keys, budgets, and usage reports; the inference plane is OpenAI API-compatible and uses only the token key issued for the group, so application code never touches account credentials.

This separation of responsibilities not only improves security but also decouples policy configuration from business logic. You can centrally define model access permissions, retry rules, and fallback chains in the control plane, while the inference plane remains simple and focused on request handling.

Defining Fallback Policies in a Token Group

The core of the policy is the token group. In the example, we define two allowed model aliases: Kimi-K3 as the primary model and GLM-5.3-Flash as the fallback model, and set up the fallback relationship. The retry policy includes the maximum number of attempts, retry delay, and exponential backoff.

allowed_models = ["Kimi-K3", "GLM-5.3-Flash"]
fallbacks = {"Kimi-K3": ["GLM-5.3-Flash"]}
retries = {
    "max_attempts": 2,
    "retry_delay_ms": 250,
    "backoff": "exponential",
}
group = client.llm_token_gateway.token_groups.create(
    name="gateway-fallbacks-demo",
    allowed_models=allowed_models,
    fallbacks=fallbacks,
    retries=retries,
    max_budget=10,
    budget_duration="1d",
    idempotency_key=f"gateway-fallbacks-group-{uuid.uuid4()}",
).data

The group restricts which models the key can call and defines how failures propagate through the chain. The example locally validates the same policy constraints before making API requests—for instance, the fallback chain depth cannot exceed three levels, max_attempts must be between 1 and 5, the fallback chain cannot reference itself or contain duplicates, and every fallback model must appear in allowed_models. The gateway also enforces these rules; policy.py deliberately sends an invalid update with max_attempts=9 and expects an HTTP 400 without changing the group state.

Token group is a named collection of inference access rules and budgets in Telnyx AI Gateway. When creating a token group, you can declare the model aliases the group is allowed to call, the fallback relationships between models, retry parameters, and budget limits and periods. Then you issue a token key for the group; the application uses this key to access the OpenAI-compatible inference endpoint without holding account-level API keys. The token group thus serves as both a permission boundary and a policy container: the model whitelist restricts which models the key can reach, the fallback chain defines the transfer path on failure, the retry configuration uniformly controls attempt behavior, and the budget constrains the maximum spend for the group in a given period. Updating or deleting a token group is a control-plane operation requiring account credentials; the inference plane works with just the token key.

Keep Retries at the Same Layer

The key principle is that retries and model fallbacks are the gateway's responsibility, so the OpenAI client should disable its own retry behavior. The example achieves this by setting max_retries=0.

llm = OpenAI(
    base_url="https://llm.telnyx.com/v1",
    api_key=token_key,
    max_retries=0,
)

This is crucial because SDK retries are new requests layered on top of gateway attempts. Disabling them ensures that response headers, ledger records, and billing remain consistent around a single retry policy. Otherwise, you may see inflated attempt counts while the actual model invocation path becomes hard to reconstruct.

How the Gateway Handles Failures

The gateway's classification of failures reflects its fine-grained judgment of request state. Definite no-work failures, such as pre-dispatch errors, 429, and 529, can retry the same model; indeterminate failures, such as timeouts, 408, 500, 502, 503, and 504, move to the next fallback model because the original request may have already reached the provider. Context length rejections are terminal because sending the same oversized context to another attempt cannot fix the input problem.

The distinction between definite no-work failures and indeterminate failures is based on whether the request may have already produced actual computation or side effects on the provider side. Pre-dispatch errors indicate the request never left the gateway, and 429 (rate limiting) and 529 (overloaded) usually mean the provider explicitly rejected the request without starting processing, so retrying the same model is safe. In contrast, for timeouts, 408, 500, 502, 503, 504, and similar cases, the gateway cannot confirm whether the original request was received and executed by the provider; retrying the same model rashly could cause duplicate billing or duplicate side effects, so the policy chooses to move to the next fallback model. Context length rejection is a defect in the request itself—changing models won't change the fact that the input exceeds limits—so it terminates immediately. This classification directly affects billing accuracy and failover effectiveness.

Forcing Model Fallback for Demonstration

Waiting for a provider outage is not a useful test plan. The example forces fallback through a controlled race: it sends a streaming request to Kimi-K3, sets a one-second first-event deadline, and passes a heavy input of about 22,000 tokens (including a unique random number to avoid KV cache effects). The goal is for the primary model's first event to miss the deadline while GLM-5.3-Flash completes the request as the fallback.

result = chat_completion(
    token_key,
    "Kimi-K3",
    build_forced_failure_prompt(),
    extra_headers={
        "x-ltg-request-timeout": "1000",
        "x-ltg-max-attempts": "1",
    },
    stream=True,
    max_tokens=64,
)

Since provider latency varies, the script tries up to three times and honestly reports whether the primary model responded within the deadline or whether both attempts timed out.

Verifying Live Requests via Response Headers

The client reads two key response headers from the raw response: x-ltg-served-model and x-ltg-attempts. A successful failover should show:

requested model:   Kimi-K3
served model:      GLM-5.3-Flash
x-ltg-attempts:    2 or more

The SDK response model is also useful, but gateway response headers explicitly show the routing decision on the live path.

Verifying the Same Event in the Consumption Ledger

The next step is to read the gateway consumption events issued for the token key. When requested_model and model differ, the ledger provides a persistent counterpart to the response headers. The example uses half-open UTC date ranges and snapshot pagination to ensure reports are reproducible even as the event set changes.

events = client.llm_token_gateway.spend.retrieve_events(
    start_date=start_date,
    end_date=end_date,
    token_key_id=token_key_id,
    page_number=1,
    page_size=50,
)
for event in events.data:
    print(
        event.requested_model,
        "->",
        event.model,
        "attempts:",
        event.attempts,
        "status:",
        event.status,
        "cache_hit:",
        event.cache_hit,
    )

GET /usage/summary adds totals and per-model summaries when the account has the llm_token_gateway.usage.read permission. If that endpoint returns 403, it is treated as a non-fatal warning because the consumption event ledger still contains request-level proof.

Per-Request Controls, Caching, and Change Safety

The gateway supports request-level overrides for attempt counts, delays, backoff, and timeouts, allowing individual requests to be adjusted without rewriting token group policies. The example also updates cache_ttl to 60 seconds and sends the same prompt twice. Gateway caching is best-effort, so the script does not fabricate cache hits: when a repeated request is served from cache, it reports x-ltg-attempts: 0; otherwise it directs you to the ledger's cache_hit field.

Every update and delete follows a change-safety pattern: fetch the latest ETag, send it via If-Match, and include an Idempotency-Key. A stale ETag returns 412, preventing one process from overwriting another's newer configuration.

ETag and If-Match form optimistic concurrency control based on version numbers. The server generates an ETag for each resource configuration; when a client updates, it declares the version it is based on via the If-Match header. If the resource has been modified by someone else in the meantime, the ETag no longer matches, and the server returns 412 Precondition Failed, refusing the overwrite. Combined with Idempotency-Key, the same logical operation is not executed twice on retry, avoiding duplicate creation or duplicate charges caused by network flakiness. This pattern is especially important when multiple processes or operators manage gateway policies simultaneously; it turns the implicit risk of "last writer wins" into explicit version conflicts, making configuration changes predictable and auditable.

Running the Example

After cloning the repository and installing dependencies, add your Telnyx account API key to .env, then run the local smoke test. After that, run the five-step demo: provision.py creates the group and key (the key is printed only once), and after setting environment variables, run inference.py, ledger.py, policy.py, and teardown.py in sequence. The live demo incurs real inference costs; the example configures a maximum budget of $10 per day, but it is advisable to review the policy before running.

provision.py stores only the group and key IDs in .gateway_state.json and never writes the token key to disk. teardown.py revokes the token key, deletes the group, verifies that the revoked key receives HTTP 401, and deletes the local state file.

A Reusable Pattern

This useful pattern is not limited to these two models: put model access, retry rules, fallbacks, and budgets in gateway policies; give applications scoped token keys instead of account credentials; disable overlapping client retries; read routing evidence from response headers; confirm the same event in the consumption ledger. This turns model failover from an assumption into a fact that operators can inspect during a request and audit afterward.

One-line Summary

Telnyx AI Gateway centralizes retries and fallbacks at the gateway layer and makes the entire failover process observable through response headers and a consumption ledger.

Paragraph Summary

The article explains how Telnyx AI Gateway moves model retry and fallback policies from the client up to the gateway layer, using the open-source gateway-fallbacks example to demonstrate the complete flow. The core idea is separation of control and inference planes: account keys configure token groups, budgets, and usage reports, while applications hold only scoped token keys. Token groups define allowed models, fallback chains, and retry parameters, and clients must disable their own retries to avoid distorted attempt counts. The gateway distinguishes definite no-work failures from indeterminate failures, retrying the same model or moving to a fallback model accordingly. The example forces fallback by setting a first-event timeout, then verifies in real time using x-ltg-served-model and x-ltg-attempts response headers and audits the same event in the consumption ledger. The article also covers per-request overrides, caching, ETag change safety, and budget control.