Unifying Multiple LLM APIs: The Design Philosophy Behind an Open-Source Gateway

The Hidden Cost of Multi-Model Integration

Developers working with large language models (LLMs) almost universally run into the same pain point: every provider has its own request and response format, error code system, and authentication method. Switching from one model to another typically means rewriting an entire adapter layer.

One independent developer shared their solution with the community — a lightweight LLM Gateway with a straightforward goal: standardize all model calls under a single unified interface. In their own words, "every time I switched vendors, I was writing the same adapter code all over again" — and that's exactly what motivated them to build it.

What's Actually the Problem?

On the surface, every provider's API does the same thing: send a prompt, receive a completion. But the implementation details diverge significantly.

Fragmentation Across Formats and Protocols

Different providers diverge across three dimensions:

• Request/Response Schema: Field names, message structures, and parameter names vary widely — messages, prompt, and input are all used interchangeably across providers.
• Error Codes: Rate limiting, timeouts, and authentication failures are surfaced differently by each provider, making error handling logic nearly impossible to reuse.
• Authentication Setup: Some use an API key in the header, others use OAuth, and some require an additional organization ID.

None of these differences are hard to handle individually — but stacked together in a project that calls multiple models simultaneously, they snowball into a mass of repetitive, error-prone glue code.

It's worth noting that OpenAI's Chat Completions API format has effectively become the de facto industry reference — many newer providers (like Mistral and Together AI) deliberately support its /v1/chat/completions interface to lower the barrier for developers migrating away. But this "compatibility" is often only skin-deep: the implementation details of function calling, streaming output, and tool use field naming still differ subtly between providers. More problematically, Google Gemini and Anthropic Claude have chosen entirely different schema designs — Gemini uses a nested contents + parts structure, while Claude extracts the system prompt as a top-level field. This means that even integrating just three mainstream models already requires handling at least three distinct message structure transformations in your adapter layer.

The Cost of Reinventing the Wheel

When a product needs A/B testing, cost optimization, or failover across different models, adapter logic baked into business code becomes a liability. Every new provider integration requires re-validating format conversions, error mappings, and retry strategies. Maintenance costs grow linearly — or worse, non-linearly — with the number of providers.

The Core Value of a Gateway Approach

This project's approach is to insert a unified gateway layer between the application and each LLM provider. It exposes a single interface outward while internally translating standard requests into each vendor's proprietary format and normalizing responses back.

The direct benefits are:

• Switch models with zero code changes: The business layer talks to one interface; swapping the underlying model requires no changes to calling code.
• Centralized error handling: Error codes are mapped uniformly at the gateway layer, so upstream logic doesn't need provider-specific branching.
• Consolidated authentication: Each provider's keys and auth details are managed in one place, eliminating credentials scattered across the codebase.

The author emphasizes this is just a "side project" built for their own workflow — but it addresses a remarkably common engineering need. As the number of available models grows, the value of an abstraction layer only increases.

Build vs. Buy

At the end of their post, the author raised a question worth discussing: how do others handle multiple LLM APIs in their projects? Do they write their own adapters, or use existing tools?

This is a tradeoff every team has to make:

• Build your own gateway: Flexible and fully in your control, able to precisely match your business needs — but requires ongoing maintenance to keep up with each provider's API changes.
• Use existing tools: Open-source LLM proxy/routing solutions work out of the box with lower community maintenance overhead, but may introduce additional abstraction layer latency or feature bloat.

For small projects calling only two or three models, a simple homegrown gateway is often sufficient. But when the number of providers, routing strategies, and observability requirements grows complex, a mature open-source gateway becomes the saner choice.

There are already quite a few mature open-source LLM gateway and routing solutions available, each with a distinct positioning. LiteLLM is the most widely used in the community — a Python library supporting 100+ models with a unified OpenAI-compatible interface that can also be deployed as a standalone proxy service. PortKey and OpenRouter are hosted gateway services that require no self-deployment, making them ideal for rapid prototyping. Martian and Unify go further by offering intelligent routing, automatically selecting the most cost-effective model based on request characteristics. For scenarios requiring enterprise-grade observability (logging, billing, rate limiting), solutions like Kong AI Gateway layer LLM-specific functionality on top of traditional API gateway capabilities. When evaluating options, beyond feature fit, you'll also want to assess the latency impact of adding an external dependency and the risk of vendor lock-in.

Conclusion

In the multi-model era, a unified interface layer is shifting from "optional optimization" to "foundational infrastructure." This small open-source gateway is positioned as a personal tool, but it illustrates a clear trend: encapsulating provider differences within a single layer and keeping business code decoupled from specific models is a key step toward more maintainable AI applications. Whether you build your own or adopt an existing solution, planning for this abstraction layer early will save significant rework costs as you scale to more models down the road.