40 Lines of Python to Handle LLM Retries, Timeouts, and Model Fallback

Title

40 Lines of Python to Handle LLM Retries, Timeouts, and Model Fallback

Meta Description

Add hard timeouts, exponential backoff retries, and model fallback to OpenAI-compatible APIs in about 40 lines of Python, with full code breakdown, retryable status code checks, and practical notes.

Article Content

When calling LLM APIs, what breaks your prototype first is often not the prompt but 429 rate limiting, hung requests, or a model being temporarily unavailable. A member of the APIGOTO team shared a roughly 40-line Python wrapper that solves all three at once: hard timeouts, exponential backoff retries, and model fallback. Although the author states they work on that gateway project, the code itself is vendor-agnostic and works with any OpenAI-compatible endpoint.

Core idea: three layers of protection

The author breaks the problem into three steps. First, set a hard timeout for each request so workers don't get stuck on slow responses; second, use exponential backoff retries on rate limits and server errors; third, if a model keeps failing, switch to the next model in the list. The priority order is: solve within the same model via retries when possible, and only enter fallback after retries are exhausted.

This order matters. Retries are cheap and don't change output style; model fallback guarantees availability but different models may produce inconsistent outputs, so it is a last resort.

Exponential backoff is a retry strategy for transient failures: the wait time before each retry grows exponentially with the number of failures, usually with random jitter to avoid the "thundering herd" effect when many clients retry at once. Its core assumption is that rate limits and server errors are often transient, and immediate retries only worsen congestion. In the code, min(2 ** i, 8) means the i-th retry waits 2 to the power of i seconds, capped at 8 seconds, giving the server a recovery window while preventing any single request from blocking too long.

Model fallback means automatically switching to the next model in a fallback list when the primary model remains unavailable after retries are exhausted. It improves system availability at the cost of possible differences in output style, format adherence, and knowledge freshness across models, so it suits scenarios with low consistency requirements or serves as a last line of defense.

Code implementation: trade-offs within 40 lines

When creating the OpenAI client, the wrapper sets two key parameters: timeout=30 ensures requests don't wait indefinitely, and max_retries=0 disables the SDK's built-in automatic retries, handing control to its own chat function. The author defines a set of retryable status codes {429, 500, 502, 503, 504}, covering rate limits and gateway-level errors.

chat(messages, models, attempts=3) uses a double loop: the outer loop iterates over the model list, and the inner loop tries each model up to 3 times. When catching APIStatusError, it first checks the status code—if it is not in the retryable set (e.g., 400, 401, 404), it breaks because retrying is pointless and moves directly to the next model; for retryable errors and APITimeoutError or APIConnectionError, it logs the exception and continues with backoff. The backoff duration uses min(2 ** i, 8), i.e., 1 second, 2 seconds, 4 seconds, capped at 8 seconds, to avoid long blocking. If all models fail, it raises RuntimeError with the last exception information.

On success, the function returns a (model, text) tuple instead of just text, so the caller knows which model actually responded. The author specifically emphasizes logging this in the Notes: when fallback triggers frequently, you need to know immediately whether the primary model is having problems.

The OpenAI Python SDK's max_retries parameter controls the SDK's internal automatic retry count for connection errors, timeouts, and 5xx/429 responses, typically defaulting to 2. Setting it to 0 disables this built-in retry layer, leaving all retry logic to business code. The benefit is avoiding "double backoff"—SDK retries stacking with custom retries leading to unpredictable wait times—and making it easier to uniformly log retry counts and final failure reasons. The cost is having to handle transient failures that the SDK would otherwise cover.

Common pitfalls

The author offers several practical tips. Use the actual model IDs listed by the provider (verify via client.models.list()), don't guess; 401 or 404 means the key or model name is wrong, and retrying won't fix it; because it relies only on base_url and the API key, the same code can point to OpenAI official, self-hosted services, or a gateway.

The real value of such a wrapper is separating "availability" from frontend business logic. During prototyping, it handles most of the first wave of production failures; once traffic grows, you can layer on circuit breakers, metrics, and alerts. The licensing notice also reminds readers: the example's default base_url points to the author's team's gateway, but you can change it to any compatible endpoint and it runs independently without affecting the code's validity.

Applicability boundaries

Note that model fallback addresses "availability", not "consistency". If downstream systems have strict requirements on output format or style, switching models may add validation costs. Also, messages may be duplicated during retries, so watch out in billing scenarios; the author does not discuss idempotency or request deduplication, which users need to add based on their own scenarios.

One-line Summary

A ~40-line Python wrapper that improves the availability of OpenAI-compatible APIs with three layers of protection: hard timeouts, exponential backoff retries, and model fallback.

Paragraph Summary

This article introduces a ~40-line Python wrapper that addresses common issues when calling OpenAI-compatible LLM APIs: 429 rate limits, hung requests, and model unavailability. Its core is three layers of protection: setting a hard timeout per request, exponential backoff retries for retryable errors, and switching to fallback models after retries are exhausted. The code disables the SDK's built-in retries and lets a custom function take control, retrying only on 429/500/502/503/504 while directly switching models on 400/401/404. The function returns the actual responding model name for monitoring fallback frequency. The author also reminds readers to use actual provider model IDs, consider fallback's impact on output consistency, and watch for duplicate billing from retries.