Free LLM API Returning 429? A Practical Troubleshooting and Multi-Source Routing Handbook

Every free LLM API acts generous right up until you hit the limit — and then an HTTP 429 cuts your demo or batch job in half. Most tutorials stop at "just add a retry loop," but when you're engineering on free tiers, what actually works is breaking the problem down by provider and by layer.

First, figure out which limit you hit

429 isn't one problem — it's four. Free tiers usually meter along several dimensions at once, and the fix depends on which one was triggered:

• RPM (requests per minute): You're sending requests too fast. Smoothing your send rate works; adding capacity doesn't.
• TPM (tokens per minute): A few long prompts eat up a whole minute's quota. You need to shorten context and cap max_tokens.
• RPD (requests per day): The day's quota is gone. There's no code-level fix — you have to spread the work across multiple days or multiple providers.
• Concurrency: The tier allows only one in-flight request, and you sent two. Use a queue, not parallelism.

Before you touch code, read the response body and headers. Groq returns x-ratelimit-remaining-requests (per day) and x-ratelimit-remaining-tokens (per minute), plus retry-after — the 429 itself tells you which window is empty and when it comes back. The Gemini API returns 429 RESOURCE_EXHAUSTED, points to the violated window in details[].quotaId, and provides a retryDelay for per-minute limits; its daily quota resets at midnight Pacific Time. For OpenRouter free models, the daily cap jumps from a small allowance to a bigger one after the account purchases credits, and the error text plus the x-ratelimit-reset-requests header tell you which window closed. Some providers give you nothing: if the 429 is bare, treat all windows as closed and back off on an hourly scale.

HTTP 429 and rate-limiting semantics: 429 is the "Too Many Requests" status code defined in RFC 6585, and it typically appears at the API gateway or server-side throttling layer, before business logic runs. It isn't an error — it's a flow-control signal meaning "your request is valid, but it exceeded your quota for the current window." Semantically, 429 is a retryable status code, fundamentally different from 401 (authentication failure) or 403 (insufficient permission) — the latter two are useless to retry, while a 429 may naturally recover once the window refreshes. So the correct reaction to a 429 isn't to immediately swap keys or providers — it's to first determine which dimension was violated. Most providers tell you which quota ran out via Retry-After, X-RateLimit-* headers, or structured fields in the response body (like quotaId). Ignoring these clues and blindly retrying often amplifies consumption, making a window that was seconds away from recovering even worse.

Retry like you mean it

Three rules cover most cases:

1. Follow retry-after when it's present. The provider is telling you the truth; guessing the timing will only make things worse.
2. When it's absent, use exponential backoff with jitter: 1s, 2s, 5s, 10s, plus a random offset, capped at a few minutes. Looping on a daily cap is pointless — after 3 to 5 attempts, stop and throw an error.
3. Count tokens, not requests. A retry loop tuned for RPM will still fail when TPM is the real bottleneck, because the retries themselves burn more tokens every minute.

Exponential backoff and jitter: Exponential backoff means the wait before each retry grows by a fixed multiplier (usually 2), e.g. 1s, 2s, 4s, 8s, until it reaches a cap. Its purpose is to stretch the time scale between a throttled server and the client, avoiding a "retry storm" under high concurrency. Exponential backoff alone isn't enough: if many clients get rejected at the same time, they'll retry in sync at the same backoff moments, creating periodic spikes. Jitter adds a random offset to each wait — commonly a random value in the range [0, delay] or [delay/2, delay] — to spread out the retries of the same batch of clients. Combined with a cap (say a few minutes) and a maximum retry count (say 3 to 5), backoff becomes genuinely usable. Note that backoff only helps for short windows like RPM, TPM, and concurrency; for a quota that zeroes out for the day, like RPD, any backoff is futile — you should fail fast or switch providers.

Don't pin all your hopes on one well

A single provider's free tier is just one quota pool. The structural fix is to treat the free layer as "a pool of pools":

• Hold multiple OpenAI-compatible keys at once (Gemini, Groq, NVIDIA NIM, OpenRouter, Cloudflare Workers AI, Cerebras, etc. each publish their own limits). When one window closes, fail the request over to the next provider. Since they all speak the OpenAI chat format, your client code doesn't change — just the base URL and key.
• Route by limit shape, not just availability. Overnight summarization jobs need RPD headroom (Groq's daily quota or Gemini's daily cap); interactive prototyping needs RPM smoothness; one-off extraction can safely burn through a token-limited tier.
• Watch the terms. Some free tiers (Cohere's trial key, Mistral's La Plateforme free tier) are explicitly for evaluation only, not production; OpenRouter's terms prohibit reselling API access. Read carefully before you build a product on top.

Hand-rolling this layer means provider abstraction, per-provider window tracking, queues, and health checks for each one — most of the work of a standalone project. If you don't want to maintain that layer yourself, there's a ready-made router: FreeLLMAPI, which the author is building, is an open-source, self-hosted router that holds your provider keys, handles 429s and retries with backoff, falls back to other providers and models, and exposes a single OpenAI-compatible endpoint. The same work can be done by hand with the handbook above — the router just lets you write it once and self-host it.

OpenAI-compatible interface and provider abstraction: The so-called OpenAI chat format refers to a JSON structure where the request body looks like {model, messages: [{role, content}]} and the response body looks like {choices: [{message}]}. Providers like Gemini, Groq, NVIDIA NIM, OpenRouter, and Cerebras mostly offer compatible endpoints in this format — which is exactly what makes multi-provider failover "just swap the base URL and key" possible. But compatible doesn't mean identical: model names, tool/function calling fields, streaming chunk details, and error object structures often differ. So in real projects, it's better to maintain a per-provider adapter config on top of the compatibility layer (model name mappings, timeouts, rate-limit parameters, feature support), rather than assuming all endpoints behave the same. That way you can safely switch when one provider is throttled or misbehaving, without introducing new parsing or behavior bugs.

Log the shape of your traffic

Whichever path you pick, log per request: provider, model, latency, status, and rate-limit headers. A week of logs will tell you which window you're actually hitting — and that single fact determines whether you need smoother sending, shorter prompts, a second provider, or just a $5 paid tier. Most "we need more quota" problems turn out to be "one long prompt running in a loop." A 429 is information — read it first, then go regenerate your key.

One-line Summary

Break down free LLM API 429s into RPM/TPM/RPD/concurrency, retry per dimension, and do multi-provider failover.

Paragraph Summary

The article points out that HTTP 429 from free LLM APIs isn't a single problem but the result of four distinct limits — RPM, TPM, RPD, and concurrency — each requiring a different fix. The author advises reading the response body and headers first — for example Groq's x-ratelimit-* and retry-after, Gemini's quotaId and retryDelay, OpenRouter's window hints — and then treating the right cause. On retry strategy, the core is to honor retry-after, use exponential backoff with jitter and a retry cap, and optimize by tokens rather than requests. The structural solution is to treat free tiers as "a pool of pools": hold multiple OpenAI-compatible keys, fail over to the next provider when one window closes, and route tasks by the "shape" of the limit. Finally, the author stresses logging per-request provider, model, latency, status, and rate-limit headers — a week of logs will pinpoint the real bottleneck.