One API: An Open-Source API Gateway Managing 30+ LLMs Through a Single Interface

Project Overview: Why You Need One API

In today's era of flourishing large language models, developers and enterprises often need to integrate with multiple LLM providers simultaneously—OpenAI, Anthropic Claude, Google Gemini, DeepSeek, Tongyi Qianwen, Wenxin Yiyan… Each has its own API format, authentication method, and billing logic, causing management costs to skyrocket.

One API (GitHub: songquanpeng/one-api) is an open-source project born to solve this exact pain point. It exposes a unified OpenAI-compatible interface to the outside world while intelligently routing requests on the backend to over 30 mainstream LLM providers, along with comprehensive Key management, quota control, and redistribution capabilities. From an architectural perspective, One API is essentially a vertical API gateway purpose-built for LLM scenarios. An API Gateway is a classic pattern in microservices architecture, responsible for cross-cutting concerns like request routing, authentication, rate limiting, and circuit breaking—Kong, Envoy, and APISIX are all well-known general-purpose API gateway solutions. One API builds on this foundation with deep adaptations for LLM-specific business logic—token-based billing, Server-Sent Events (SSE) passthrough, multi-model protocol translation, and more—making it a specialized middleware connecting the application layer to LLM service layers.

The project has earned over 32,800 Stars on GitHub with more than 6,200 forks, making it one of the most popular open-source LLM API gateway solutions available today.

Core Features in Detail

Unified API Adaptation Layer: One Interface for All LLMs

One API's most core value lies in protocol unification. Regardless of whether the backend connects to Azure OpenAI, Anthropic Claude, Google Gemini, DeepSeek, ByteDance Doubao, ChatGLM, iFlytek Spark, Tongyi Qianwen, 360 Zhinao, or Tencent Hunyuan, the externally exposed interface is always the standard OpenAI API format.

The "OpenAI API format" specifically refers to the Chat Completions API specification centered on the /v1/chat/completions endpoint, using a JSON-formatted messages array to convey conversation context. Thanks to ChatGPT's explosive growth, OpenAI's API format has become the de facto standard for LLM invocations—virtually all subsequent model providers offer an OpenAI-compatible mode, which laid the foundation for One API's protocol unification strategy. This standardization trend is similar to how REST APIs replaced SOAP as the mainstream web service protocol in earlier years: once an interface paradigm gains sufficient ecosystem momentum, newcomers tend to actively maintain compatibility rather than start from scratch, in order to reduce developer migration costs. On top of this, One API further handles differences across providers in authentication header formats (e.g., OpenAI uses Authorization: Bearer while some domestic providers use custom headers), error code definitions, and streaming data chunk structures, truly achieving "integrate once, use everywhere."

This design brings several direct benefits:

• One SDK for everything: Frontend applications only need to integrate once and can seamlessly switch underlying models
• Seamless ecosystem compatibility: OpenAI ecosystem tools like ChatGPT-Next-Web, LobeChat, and Chatbox can connect directly without any modifications
• Near-zero migration cost: Model switching and A/B testing only require changing channel configuration in the backend—business code remains completely untouched

Key Management and Redistribution

One API includes a complete multi-tenant Key management system, which is the key differentiator from simple reverse proxy tools.

Multi-tenancy is a core concept in SaaS architecture, referring to a single system simultaneously serving multiple isolated users or organizations. In the LLM API management context, multi-tenancy means different teams, projects, or customers can share the same infrastructure while each having independent access credentials, usage quotas, and billing records. In contrast, a "single-tenant" model—where each user has a dedicated system instance—offers stronger isolation but multiplies operational costs. One API employs a logical isolation multi-tenant approach: all tenants share the same database and service process, with data isolation achieved at the application layer through tokens and user IDs. One API's multi-tenant design manifests in the following aspects:

• Channel management: Configure multiple upstream API Keys with support for load balancing by priority and weight, with automatic failover to backup channels when one encounters errors
• Token management: Generate independent access tokens for different users or applications, with individual quota limits and expiration times
• Usage tracking: Detailed records of each token's call count, token consumption, and cost statistics—billing at a glance

A few more words about token-based billing: LLM billing is typically based on tokens—tokens are the smallest units the model processes text in. They don't equate to "characters" or "words" but are subword segments split by a tokenizer based on statistical patterns in training data. In English, each word corresponds to roughly 1-1.5 tokens; in Chinese, each character typically maps to 1-2 tokens (since Chinese characters are often split into multiple byte combinations under BPE and similar tokenization algorithms). Token pricing varies dramatically across models—for example, GPT-4o's input price is several times that of GPT-3.5-turbo, while domestic models like DeepSeek often offer more competitive pricing—DeepSeek-V3's input price is approximately 1/30th of GPT-4o's. Additionally, LLMs typically charge separately for input (Prompt) and output (Completion), with output token prices usually 2-4x higher than input due to greater computational overhead during generation. One API's quota system converts the different pricing structures from various upstream providers into a unified internal quota unit (by default converting USD to internal quota at a 1:500000 ratio), enabling cross-model cost accounting so administrators can clearly track overall AI spending.

This mechanism is particularly suited for sharing API resources within teams, providing API resale services externally, or allocating limited quotas to students in educational settings.

Load Balancing and Automatic Failover

One API supports configuring multiple channels for the same model, distributing traffic by priority and weight. When a channel returns an error or times out, the system automatically forwards the request to the next available channel—completely transparent to the caller.

Load balancing and failover are two cornerstone technologies for ensuring high availability in distributed systems. Load balancing distributes requests across multiple backend nodes to prevent single-point overload, with common strategies including Round Robin, Weighted Round Robin, and Least Connections. One API employs a priority + weight scheduling model—it preferentially uses high-priority channels, with traffic distributed by weight ratio among channels of the same priority. This two-tier scheduling strategy is highly flexible in practice: for example, you can set the lowest-priced channel as highest priority, set the most stable but more expensive channel as secondary priority for fallback, and use weights for fine-grained traffic distribution within the same priority level. Failover means that when a primary node becomes unavailable, the system automatically switches traffic to backup nodes without manual intervention.

In the LLM API context, failover mechanisms are especially important: LLM providers' APIs frequently experience rate limiting, service degradation, or temporary outages, especially during new model launches or traffic peaks. Rate limiting is a common mechanism for API providers to protect backend resources, typically manifested as returning HTTP 429 (Too Many Requests) status codes with a Retry-After header indicating how long the client should wait. Different providers have vastly different rate limiting strategies: OpenAI limits by both requests per minute and tokens per minute, while some domestic providers limit by concurrent connections. One API detects channel health by monitoring HTTP status codes (such as 429, 500, 502, 503) and timeout signals, achieving sub-second automatic switching. For example, if you configure both DeepSeek and Tongyi Qianwen simultaneously, when one goes down the other automatically takes over, ensuring service continuity.

Supported Model Providers

As of now, mainstream LLM providers adapted by One API include but are not limited to:

Category	Providers
International	OpenAI, Azure OpenAI, Anthropic Claude, Google Gemini, Cohere
Chinese Domestic	DeepSeek, ByteDance Doubao, Baidu Wenxin Yiyan, Alibaba Tongyi Qianwen, iFlytek Spark, Tencent Hunyuan, 360 Zhinao, Zhipu ChatGLM
Relay & Aggregation	Cloudflare Workers AI, OpenRouter, Custom Channels

Notably, while Azure OpenAI and direct OpenAI share the same underlying models, they differ significantly in API endpoint format, authentication method, and deployment model—Azure uses a resource group + deployment name URL structure, authenticates via Azure AD or API Key, and requires models to be manually deployed in Azure Portal before they can be called. One API transparently encapsulates these differences—users simply fill in their Azure endpoint and key to call it just like the original OpenAI.

This broad compatibility makes One API the de facto LLM API middleware standard. Regardless of which model provider an enterprise chooses, they can manage it uniformly through One API, and adding or removing providers later only requires adding channels in the backend.

One API Deployment Tutorial: One Docker Command

One API's backend is written in Go, with a React frontend, compiled into a single executable file, with an official Docker image also provided.

Go (Golang) is a programming language released by Google in 2009, renowned for its fast compilation speed, excellent concurrency model (based on lightweight goroutines), and simple deployment. Go compiles to a statically-linked single binary that doesn't depend on external runtime environments (unlike Java requiring a JVM or Python requiring an interpreter), making deployment extremely convenient—just copy one file to the server and run it. Go's goroutine mechanism is particularly well-suited for API gateway scenarios: each goroutine occupies only about 2-8KB of stack memory (compared to 1-8MB for operating system threads), meaning a single Go process can easily maintain tens of thousands of concurrent connections—a critical advantage for LLM gateways that need to handle large numbers of simultaneous streaming long connections. This characteristic provides clear advantages in containerized scenarios and is an important reason why cloud-native infrastructure tools like Docker, Kubernetes, and Terraform chose Go. One API inherits this advantage with a small Docker image (typically 50-100MB), fast startup (sub-second cold start), low memory footprint (approximately 30-50MB idle), and the ability to handle considerable concurrent request volumes on a single instance.

The deployment process is very simple:

docker run -d --name one-api \
  -p 3000:3000 \
  -e TZ=Asia/Shanghai \
  -v /home/ubuntu/data/one-api:/data \
  justsong/one-api

After startup, visit http://localhost:3000 to access the admin dashboard. It uses SQLite for data storage by default, requiring no additional database installation. SQLite is an embedded database engine that stores data in a single file without needing a separate database service process, making it ideal for small-to-medium deployments and quick validation scenarios. Despite being lightweight, SQLite's performance is respectable—in read-heavy, write-light scenarios, single-machine QPS can reach tens of thousands. However, SQLite uses file-level locking for writes, which can become a bottleneck in high-concurrency write scenarios (such as heavy simultaneous API calls generating log records). Therefore, production environments also support switching to MySQL or PostgreSQL by specifying a connection string via the SQL_DSN environment variable—truly plug-and-play.

The basic configuration workflow after deployment:

1. Add channels: Enter API Keys from various LLM providers in the backend and select the corresponding provider type
2. Create tokens: Generate access tokens for yourself or team members, setting quotas and expiration dates
3. Connect applications: Fill in One API's address and token in your application or tool, replacing the original OpenAI API address

Technical Note on Streaming Responses

It's worth mentioning that a key feature of LLM APIs is support for streaming responses—instead of waiting for the entire generation to complete before returning, the model pushes generated text token-by-token to the client in real time. This mechanism is implemented via the Server-Sent Events (SSE) protocol—the server continuously sends event streams prefixed with data: over an HTTP long connection, which the client parses line by line and renders in real time. SSE is part of the HTML5 specification and, compared to WebSocket, is unidirectional (server-to-client only), based on standard HTTP protocol, and natively supports reconnection—making it perfectly suited for the "request once, output continuously" interaction pattern of LLMs. In OpenAI's implementation, each streaming data chunk is a JSON object containing incremental text in the choices[0].delta.content field, with data: [DONE] marking the end of the stream.

Streaming responses are crucial for user experience: a GPT-4-level long answer may take 10-30 seconds to fully generate, and if users had to wait for completion before seeing anything, the experience would be terrible. Streaming output lets users see the response begin appearing after the first token is generated (typically within 0.5-2 seconds)—this metric is called TTFT (Time To First Token) and is one of the key indicators for measuring LLM service responsiveness.

As a middleware layer, One API needs to correctly handle SSE passthrough, including keeping long connections from being unexpectedly terminated by intermediate proxies (like Nginx), correctly forwarding streaming data chunks in various formats from different providers, and accurately tracking token consumption in streaming mode. Token counting in streaming mode is a technical challenge: since text arrives chunk by chunk, One API needs to aggregate all chunks after the stream ends to calculate actual token consumption, or rely on the usage field returned by the upstream API in the final data chunk (not all providers support this). If you're using Nginx as a reverse proxy in your deployment, note that you should add proxy_buffering off, chunked_transfer_encoding on, and appropriate timeout configurations (recommend setting proxy_read_timeout to 300 seconds or more) to ensure streaming responses work properly.

Typical Use Cases

Enterprise Internal AI Platform

Enterprise IT departments can deploy One API as an internal AI gateway, centrally procuring and managing API Keys from various LLM providers while distributing independent tokens and quotas to different business departments. This achieves controllable costs and traceable usage while avoiding the security risks of API Keys scattered across various projects. In real enterprise scenarios, API Key leakage is a high-frequency security incident—developers accidentally committing keys to public GitHub repositories, embedding them in frontend code, or sharing them in plaintext via Slack are all common occurrences. Through One API's token mechanism, the actual upstream API Keys are stored only on the One API server, while secondary tokens generated by One API are distributed to teams—even if leaked, they can be revoked at any time without affecting other users, significantly reducing the risk surface of key management.

API Resale and Distribution Platform

For AI service intermediaries, One API provides ready-made multi-tenant management and billing infrastructure to quickly build API distribution platforms with prepaid quota models, eliminating the substantial work of developing an admin backend from scratch. The business model for such platforms typically involves bulk purchasing upstream API quotas at a discount, then retailing them to small and medium developers at slightly above cost—earning the margin while providing users with more convenient access and more flexible payment options.

Personal Developer Multi-Channel Aggregation

Individual developers can aggregate free trial quotas or low-cost channels from multiple platforms into One API, using automatic failover and load balancing mechanisms to obtain more stable LLM API service at lower cost. For example, you could simultaneously configure DeepSeek (cost-effective but occasionally rate-limited), Alibaba Tongyi Qianwen (has free quota), and OpenAI (highest quality but expensive) as three channels, setting DeepSeek as highest priority, Tongyi Qianwen as secondary, and OpenAI as fallback—achieving optimal balance between cost and quality.

Community Ecosystem and Project Activity

The 32,800+ Star count clearly demonstrates community recognition of this project. One API is primarily developed by songquanpeng with numerous community contributors, and issues and PRs receive relatively timely responses. Several enhanced versions have emerged around One API (such as new-api, one-api-plus, etc.), forming an active small ecosystem.

This Fork-and-Enhance pattern is very common in open-source communities (similar to Nextcloud vs. ownCloud, MariaDB vs. MySQL), driving feature innovation while promoting continuous improvement of the original project through competition. Among these, new-api is one of the most well-known derivative projects, adding a more polished UI (rebuilt with Material Design style), more granular model pricing configuration (supporting per-model rate multipliers), Midjourney image generation interface support, and data dashboards on top of the original. Around the One API ecosystem, dedicated monitoring panels (such as Grafana-based usage visualization solutions), usage analysis tools, and notification plugins integrating with WeChat Work/DingTalk have also emerged, further enriching its practicality in enterprise scenarios.

However, it's also important to note that as an open-source project, One API may require additional consideration in enterprise-grade scenarios:

• High-availability deployment: The default single-instance deployment should be supplemented with an external database and load balancer for production environments. When necessary, multiple One API instances can be deployed behind Nginx or cloud provider SLB (Server Load Balancer) for traffic distribution. Since One API itself is stateless (state is stored in the external database), horizontal scaling is relatively straightforward, but session consistency and cache synchronization need attention
• Security auditing: Since it involves API Key storage and forwarding, deployment on an internal network or with HTTPS and access controls is recommended. Key areas of concern include how API Keys are stored in the database (whether encrypted and the strength of the encryption algorithm), TLS protection during transmission, and access permission controls for the admin dashboard. Enabling One API's IP whitelist feature and regularly rotating upstream API Keys is recommended
• Compliance: Certain industries (such as finance and healthcare) have strict requirements for data flow paths, requiring evaluation of whether forwarding API requests through a middleware layer complies with relevant data security and privacy protection regulations. Particularly in scenarios involving personal privacy data (such as patient medical records or user conversation logs), you need to confirm whether One API persists request/response content locally, and whether data passing through the middleware layer violates data residency requirements

Conclusion: Who Should Use One API

One API precisely addresses the core pain point of fragmented API management in the LLM era. With extremely low deployment barriers, broad model compatibility, and practical Key management features, it has become critical infrastructure connecting the application layer to LLM service layers.

If you fall into any of the following categories, One API is worth serious consideration:

• Using two or more LLM providers simultaneously and needing unified API Key management
• Wanting to allocate independent API quotas and permissions to team members
• Building AI applications and wanting to retain the flexibility to switch underlying models at any time
• Needing a lightweight LLM API gateway without introducing heavyweight commercial solutions

It takes just one Docker command to get it running—why not give it a try.

Key Takeaways

• One API adapts 30+ mainstream LLM providers through a unified OpenAI-compatible interface, eliminating the complexity of multi-platform API integration
• Provides complete Key management, quota control, and redistribution capabilities, supporting multi-tenant scenarios
• Lightweight deployment with a single executable + Docker image, ready out of the box
• 32,800+ GitHub Stars, making it one of the most popular open-source LLM API gateway solutions
• Applicable to enterprise internal AI platforms, API resale distribution, personal developer multi-channel aggregation, and other typical scenarios