One API: Deployment Tutorial & Deep Dive into the Open-Source API Gateway for Managing 30+ LLMs

What Is One API

One API is an open-source LLM API management and distribution system created by developer songquanpeng. With over 32,800 Stars on GitHub, it's one of the most popular large model API gateway projects available today.

The problem it solves is straightforward: as the number of large language models on the market keeps growing, each with its own API format and authentication method, developers are exhausted from integrating them one by one. One API converts dozens of mainstream LLM APIs into a unified OpenAI-compatible format, allowing developers to call any model through a single interface.

From an architectural perspective, One API is essentially an API Gateway purpose-built for the LLM landscape. The API Gateway is a classic design pattern in microservices architecture, originally popularized by companies like Netflix and Amazon in large-scale distributed systems. The core idea is to place a unified entry point between clients and backend services, handling cross-cutting concerns such as request routing, protocol translation, authentication, and rate limiting/circuit breaking. In the LLM space, the value of an API gateway is amplified further—different providers have varying authentication methods (API Keys, OAuth, signature mechanisms), request formats (JSON Schema differences), and billing models (per-token, per-request, time-based). The gateway layer can completely abstract away these differences, letting developers focus on business logic.

It's worth noting that the billing unit "Token" in LLM APIs is not simply equivalent to a character or word. A Token is the basic unit that a model's Tokenizer uses to segment text. In English, one Token corresponds to approximately 0.75 words; in Chinese, due to higher character density, typically 1-2 Chinese characters map to one Token. Different providers use different tokenization algorithms (e.g., OpenAI's tiktoken, Google's SentencePiece), meaning the same text may consume different numbers of Tokens across platforms. When unifying billing and tracking, One API handles this heterogeneity by relying on the actual usage field returned by each provider rather than local estimates, ensuring billing accuracy.

Core Features Explained

Unified Adaptation for 30+ LLMs

One API covers virtually all major model providers in the current AI landscape:

• International providers: OpenAI (GPT series), Anthropic Claude, Google Gemini, Azure OpenAI
• Chinese providers: DeepSeek, ByteDance Doubao, Baidu ERNIE Bot, Alibaba Qwen, iFlytek Spark, Tencent Hunyuan, Zhipu ChatGLM, 360 AI Brain

Regardless of how different the underlying models' interface specifications are, after processing through One API's adaptation layer, they are all exposed externally in OpenAI API format. Any application compatible with the OpenAI SDK can connect directly, and switching the underlying model requires zero code changes.

Why has the OpenAI format become the unified standard? There's an interesting evolution behind this. OpenAI's Chat Completions API wasn't always the industry standard—when GPT-3 launched in 2020, OpenAI initially offered a Completions (text completion) endpoint that took a single prompt string as input. After ChatGPT went viral in late 2022, OpenAI officially released the Chat Completions API in March 2023, centered around a multi-turn conversation messages array, which better matched real dialogue scenarios. Major frameworks like LangChain and LlamaIndex quickly adopted it as their default interface, and later entrants like Anthropic and Mistral also provided compatibility layers. This "winner-takes-all" standardization effect transformed the OpenAI format from one company's proprietary protocol into the industry's infrastructure standard in just two years. This specification defines the structure of parameters like model, messages, and temperature in request bodies, as well as response fields like choices and usage—virtually all third-party tools and frameworks prioritize compatibility with this format. Therefore, "OpenAI-compatible format" means seamless integration with the entire AI development ecosystem, which is the fundamental reason One API chose it as its unified output format.

API Key Management & Redistribution

One API includes a comprehensive Key management system—a key capability that distinguishes it from simple proxy tools:

• Multi-channel configuration: Connect multiple Keys from multiple providers simultaneously, with automatic load balancing and failover
• Token management: Create sub-tokens with individual quota limits, expiration dates, IP whitelists, and other policies
• Usage tracking: Detailed records of call counts, Token consumption, and cost breakdowns for each token

This mechanism is extremely practical in real-world use—whether distributing API quotas within a team or providing billed AI services externally, it works out of the box.

The multi-channel load balancing and failover mechanism draws on design principles from traditional reverse proxies (like Nginx and HAProxy), but is customized for the characteristics of LLM APIs. Traditional load balancing primarily focuses on request latency and server load, while an LLM API gateway must also consider remaining Token quotas, provider Rate Limits, and response quality across different models.

Provider rate limits typically apply across multiple dimensions simultaneously: Requests Per Minute (RPM), Tokens Per Minute (TPM), and Tokens Per Day (TPD). When any dimension hits its cap, the API returns an HTTP 429 error (Too Many Requests), with the response header typically including a Retry-After field indicating how many seconds the client should wait. One API's failover mechanism, upon detecting a 429 response, not only immediately switches to a backup channel but also marks the rate-limited channel as "cooling down," reducing its routing weight for a period to avoid repeatedly triggering the same channel's limits. This combination of Exponential Backoff and channel cooling is the standard approach for production-grade API gateways handling rate-limiting issues, ensuring that when a provider returns 429, 503, or similar errors, requests are automatically routed to backup channels—completely transparent to the caller. This design is critical in production environments because even top-tier providers like OpenAI experience periodic service degradation.

Lightweight Deployment

One API has a very low deployment barrier, essentially offering an out-of-the-box experience:

• Single executable: After compilation, it's just one binary file with no additional runtime dependencies
• One-click Docker startup: Official Docker images are provided, with a single command to launch