Palimpsest Gateway: Open-Source LLM Key Pool Gateway with Automatic Failover

Almost every developer building LLM applications has hit the same wall: you've got a handful of API keys from different providers — two OpenAI free-tier keys, one paid, one Anthropic, one Gemini — and you discover that every application needs its own logic to handle rate limits and quota exhaustion. The moment one key hits its ceiling, the app goes down, unless you've duplicated failover logic in every single application. Keys are scattered across .env files, shared insecurely, and rotating one risks breaking something else.

One developer thought through this problem and concluded: this isn't an application-layer problem — it's an infrastructure problem. The result is Palimpsest Gateway — a self-hosted, OpenAI-compatible gateway that sits in front of all your LLM keys, pools them centrally, handles automatic failover, and maintains conversation continuity across key switches. The project is open-sourced under the MIT license, built with Python/FastAPI, and runs with a single docker compose up.

The Problem: Keys Exhaust, Apps Break

Anyone who's scaled an LLM application beyond a hobby project knows these pain points intimately. Free-tier keys hitting 429s bring apps down — and even if you catch the exception and sleep-retry, your users have already left. Monthly spend caps or daily free quotas running out come with no warning, just a cold 402 or 429. Then there's key sprawl — 5 apps times 3 providers equals 15 keys scattered across 15 .env files, where rotating one key means a full deployment.

Beyond the chaos, there's a complete lack of observability. You can't tell which key served which request, how much it cost, or why a particular request failed. What the author wanted was simple: a single unified endpoint that accepts standard OpenAI client calls (zero code changes), pools all provider keys behind the scenes, switches transparently on rate limits or errors, preserves conversation context so users don't notice the switch, runs on your own hardware with keys encrypted at rest, and comes with a dashboard that gives you the full picture.

Architecture: What's Actually Running

The core system consists of the gateway itself plus Redis. Applications (OpenAI clients) send requests to Palimpsest Gateway, which handles authentication and admission, key selection, the failover loop, session memory, usage metering, and encrypted storage — then forwards to OpenAI, Anthropic, Google Gemini, or OpenRouter in each provider's native format. Redis handles the key registry, session history, usage counters, and audit logs.

The request flow is clean and tightly connected: the application sends a gateway key (pgw_live_...) and never touches the real provider keys; the gateway checks per-user daily quotas and pool capacity for admission; if an X-Session-ID header is present, it pulls historical context from storage so the application only needs to send new messages; it then selects from active keys serving the target model, preferring rate-limited keys that still have remaining quota; it forwards the call in the provider's native format; if it encounters a rate limit, outage, or bad key, it marks that key as sleeping/exhausted/suspended and retries on the next key (same call, context carried along); finally it returns an OpenAI-format response, saves the conversation turn, and counts usage exactly once.

The Real Hard Parts and How They're Solved

Streaming Failover Without Breaking the Client

This is the trickiest part of the entire project. When a provider returns a streaming response, you can't simply "retry on another key" — the client is already receiving tokens. The author's solution: the gateway fully buffers the entire provider stream in memory before sending anything to the client. If the provider fails mid-stream, the gateway retries on the next key with the complete context (system prompt + last N turns + current message). The client always sees a single, complete, valid OpenAI stream.

The core insight in one sentence: buffer the entire stream and complete failover before any bytes leave the gateway — the client never receives a partial response. The failover loop classifies errors: transient errors mark the key for cooldown and move to the next, exhausted marks and moves on, rejected suspends the key, and only fatal errors actually raise an exception.

Cross-Provider Context Continuity

When failover happens, the replacement provider needs the conversation history — but each provider has different context window sizes and message formats. The gateway stores each conversation turn in Redis by session ID. On failover, it reconstructs a "verbatim window" — defaulting to the last 4 turns sent as-is, plus the system prompt and current message. This fits within any reasonable context window while preserving the conversational thread.

Encrypted Key Storage That Survives Restarts

Provider keys are the crown jewels — they must be encrypted at rest, never logged, never returned after creation, but the gateway needs to decrypt them at request time. The solution: a master key (Fernet) is generated on first startup and persisted to a Docker volume. Every provider key is encrypted with it before being written to Redis, and the master key never leaves the container. Rotating the master key is a destructive operation (make rotate-secrets) — this is intentional.

Key Selection That Respects Your Intent

With both free keys (daily quota limits) and paid keys (unlimited), you naturally want to exhaust free quota first, with paid as fallback. The selection algorithm sorts accordingly: keys with request caps that still have remaining quota come first (more remaining = higher priority), then unlimited keys; within each group, preference goes to providers that served previous turns of the same session to maintain locality. This sorting logic is simple but precisely aligned with cost intent.

Up and Running in 60 Seconds

The onboarding friction is kept to a minimum. Clone the repo and run docker compose up -d --wait. On first launch it generates the master key and operator key, persists them to a Docker volume, and starts the API at http://localhost:8000. Then add provider keys to .env (or via the dashboard), create a user in the dashboard to get a pgw_live_... key, and point any OpenAI client's base_url at the local gateway. Add an X-Session-ID header to your requests and conversation history is maintained server-side automatically.

What It Looks Like in Practice

The dashboard exposes virtually everything: health, usage, quota, reset schedule, and cooldown status for every key in the pool; utilization and failover events with full retry chains; per-account usage, active sessions, and quota consumption; cost by model/user/day (pricing is entered by you — no bundled presets); alerts for low capacity, key anomalies, and error rates with webhook support; and an audit log of every operator action, with CSV export for usage, requests, and audit data.

A typical request log makes the failover story clear: the chain field on one request shows it first hit key_openai_free_1, got rate-limited, then switched to key_openai_paid, with attempts at 2, context carried along, and a final outcome of success. Free key rate-limited → switched to paid → context preserved → success: the entire chain is visible at a glance.

Why "Palimpsest"?

A palimpsest is a manuscript page that has been written on, scraped clean, and written on again — with traces of the old text still faintly visible beneath. This is the gateway's metaphor: it layers conversation history across provider switches so that even when the underlying key changes, the new provider still sees the "old text" — the prior context. It's an apt name that points directly at the product's core value: continuity.

What's Next

The project roadmap includes: multi-gateway clustering (multiple instances behind a load balancer sharing Redis), provider quota sync (OpenRouter already supports reading real quotas rather than manually setting caps), request transformation (normalizing tool calls across providers, injecting system prompts), and good first issues reserved for community contributors.

For teams struggling with multi-key management, rate limit fallback, and cost observability, Palimpsest Gateway offers a compelling open-source reference architecture. It consolidates duplicated application-layer logic into infrastructure, and tackles the hardest-to-ignore problems in LLM gateways with three core mechanisms: buffered streaming failover, session-level context reconstruction, and encryption at rest. The project lives at github.com/hammrouni/palimpsest-gateway under the MIT license, with a prebuilt image available via docker pull ghcr.io/hammrouni/palimpsest-gateway:latest.