Integrating 20 Free LLM APIs into One: A Developer's Field Notes

Integrating 20 Free LLM APIs into One: A Developer's Field Notes

While building an edtech platform, I open-sourced a few small Node tools. One of them, llm-free-cascade, solves a problem that looks simple but turns out to be quite thorny: how to make multiple free LLM APIs behave like a single service.

The Core Goal and Its Challenges

The design intent was obvious: send a chat request to Groq, and if Groq refuses, try Gemini, then Cerebras, and so on through every free tier you have a key for. But the implementation exposed far more complexity than expected. From key management to error handling, from rate-limiting strategy to provider stability, every piece needed careful design.

What Is an LLM API Free Tier?

Mainstream model providers (OpenAI, Google, Groq, Cerebras, SambaNova, etc.) typically offer developers a certain free quota of calls—the "free tier." These quotas are usually bounded by requests per minute (RPM), tokens per minute (TPM), or requests per day (RPD), and going over returns a 429 status code. The defining trait of a free tier is that you get some inference capacity without paying, but the trade-offs are real: small quotas, strict rate limits, restricted model availability, and policies that can change at any time (for example, switching to a time-limited trial, requiring a credit card, or being discontinued outright). Because of this, combining multiple providers' free tiers into a single "virtual big pool" has become a common developer strategy for controlling cost and improving availability—and that is exactly the problem llm-free-cascade tries to solve: using a cascade to automatically switch between multiple free tiers so that callers upstream never notice the fragmentation of quotas underneath.

9 Key Lessons

1. Don't Keep Retrying the Same Key

The first version always started with the first key. Once it hit its daily limit, every request would still try it first, fail, and only then switch. The more keys you had, the slower the responses got.

The fix: rotate across keys. When a key is rate-limited, let it rest and skip it until it recovers.

2. Limits Usually Apply to a Single Model

A 429 rarely means "this key is exhausted"; it usually means "this model under this key has hit its limit." So before giving up on a provider, try other models under the same key.

new LLMCascade({
  keys: { groq: process.env.GROQ_API_KEY },
  models: { groq: ['openai/gpt-oss-120b', 'openai/gpt-oss-20b'] }, // the second model is only tried if the first is limited
});

3. Know Exactly How Long to Wait

A fixed cooldown is wrong—too long for per-minute limits, too short for per-day limits. Providers signal when you can retry, each in their own way:

• The Retry-After response header
• Messages like "try again in 2m59s"
• Wording that includes "per minute" or "per day"
• Known daily reset times (Gemini resets at midnight Pacific Time)

The cascade reads all of these signals. Per-minute limits wait roughly a minute; daily limits wait until the provider resets.

Rate Limits and the Retry-After Mechanism

A rate limit is a server's way of protecting its own resources. Common implementations include token buckets and sliding windows. When a client exceeds its quota, the server returns HTTP 429 Too Many Requests and may include a Retry-After header whose value can be a number of seconds or an HTTP date. Beyond that, many LLM providers put natural-language hints in the error message of the 429 body—things like "Rate limit reached, try again in 2m59s" or "You exceeded your current quota, please check your plan"—and some specify whether the limit is "per minute" or "per day." For the client, correctly recognizing these signals and retrying after the wait is far more efficient than blindly sleeping a fixed amount: a fixed wait may be too long for per-minute limits (wasting throughput) and too short for daily limits (hitting the wall repeatedly). A mature client should therefore parse headers and error messages, distinguish minute-level from daily-level throttling, and factor in each provider's known reset time (e.g., Google Gemini's free tier resets at midnight Pacific Time) when scheduling recovery.

4. Not Every Error Means the Provider Is Down

Error Type	How the Cascade Handles It
Prompt too long	Skip the provider for this request only
429/quota	Rest the key and model until reset
401/403	Stop using that key
404 (model removed)	Stop using that model
Timeout, 5xx	After 2 consecutive failures, skip the provider for a minute

The last row matters enormously. Without it, a single downed provider makes every request wait out the full timeout.

5. A Trick I Chose Not to Use

You could fire the same request at two providers at once and keep whichever responds faster. But on free tiers, that burns double the quota for one answer, so I dropped the idea.

6. Free Tiers Change Constantly

This year Cerebras's free tier became a 30-day trial, SambaNova started requiring a credit card, and GitHub Models shut down. Model names change even more often.

So the provider list lives in a single data file, providers.json. Each entry records when it was last checked and the data source, and warns about things like "non-commercial use only" or "requires a phone number." A small script checks each model name against the providers' live listings.

One caveat: many providers don't allow circumventing limits via multiple free accounts. Check each provider's terms.

7. Call It from Any Language

Someone filed an issue wanting to use it from their own tools just like the OpenAI API. Now you can run it as a local server:

npx llm-free-cascade serve        # http://127.0.0.1:8787/v1

Then point any OpenAI client at it, for example in Python:

from openai import OpenAI

client = OpenAI(base_url="http://127.0.0.1:8787/v1", api_key="unused")
reply = client.chat.completions.create(
    model="cascade",  # or "groq" to use only Groq
    messages=[{"role": "user", "content": "Explain photosynthesis in one sentence."}],
)
print(reply.choices[0].message.content)

It streams too. By default the server listens only on localhost, because it holds all your keys.

OpenAI-Compatible APIs and Local Proxies

The term "OpenAI-compatible API" means the server exposes the same request/response structure as OpenAI Chat Completions (/v1/chat/completions, a messages array, choices[0].message.content, and so on). Because OpenAI's SDK and ecosystem are so widely adopted, many clients, frameworks, and tools are written against this interface by default. So as long as the proxy layer implements the protocol, almost any language or tool can plug in with zero changes. llm-free-cascade serve is exactly such a local proxy: it listens on a local port, exposes an OpenAI-compatible endpoint, and translates requests into cascade calls against multiple underlying providers. With an OpenAI client pointed at the local address via base_url (Python, Node, Go, Rust all work), you transparently get multi-provider aggregation. One thing to note: this server holds all your API keys in memory, so by default it binds only to 127.0.0.1—don't expose the port to the public internet. If you really need remote access, add a reverse proxy and an auth layer.

8. Use Ollama as a Local Backup

If every free tier is busy at once, Ollama can answer from your own machine. No limits—just slower.

new LLMCascade({
  custom: {
    ollama: {
      baseUrl: 'http://localhost:11434/v1',
      model: 'llama3.2',
      apiKey: 'ollama', // any text will do, but it can't be empty
      tier: 5,          // try last
    },
  },
});

9. Visualize the Runtime State

npx llm-free-cascade keys     # which providers are missing, and where to sign up
npx llm-free-cascade status   # your providers, keys, and limits
npx llm-free-cascade probe    # quick live test of all your keys

In code, cascade.stats() shows per-provider and per-key call counts, failure counts, speed, and cooldown status.

Implementation and Ecosystem

llm-free-cascade is MIT-licensed, has zero dependencies, and runs on Node 18+. By cascading multiple free LLM APIs, it gives developers a unified interface while handling the messy logic of key rotation, model switching, error classification, and rate-limit recovery.

Project links: GitHub · npm · Guide

If you spot a model name that's gone stale, fixing it is a single-file PR. If you'd like tool calling to be supported in the server, open an issue describing your use case.