Building Your Own OpenAI-Compatible Gateway to 6 Chinese LLM Providers: Five Pitfalls Beyond the Happy Path

A self-hosted API gateway that funnels six Chinese model providers—DeepSeek, Kimi, Qwen, Doubao, ERNIE, and GLM—behind a single OpenAI-compatible endpoint sounds like an afternoon's work. The author spent two months of full-time effort, and what nearly sank the project wasn't the routing layer or the Kafka pipeline—it was token billing.

The project is called He-API, and the motivation is almost boringly simple: Chinese models are 80%–90% cheaper than GPT-4/Claude-class Western models on most workloads, but for developers outside China, actually integrating them is painful—Chinese-language docs, payment methods that don't work internationally, inconsistent auth across providers, and no unified format. So one endpoint, one key, one request shape became the project's core proposition.

What the Architecture Looks Like

The overall chain isn't complicated: client (any OpenAI SDK) → Nginx + Let's Encrypt → an api-gateway written in Go, which internally passes through a Redis-backed permission resolver, a Redis + Lua rate limiter, a model router, and per-provider adapter layers (kimi / deepseek^Ad / qwen / doubao / ernie / glm), ultimately hitting the upstream provider. Usage events are written to ClickHouse via Kafka, while billing, model, and provider metadata live in PostgreSQL.

Fifteen services run on systemd, all Go except for one Next.js admin console; middleware includes Postgres, Kafka, Redis, and ClickHouse, with 22 migration scripts all idempotent. In the author's own words: "This is the diagram everyone draws. What follows is where it actually bites."

Pitfall One: "OpenAI Compatible" Is 90% Edge Cases

The author knocked out the happy path for a non-streaming completion in one afternoon. Everything after that went into the long tail.

Streaming output was the first hurdle. Providers don't agree on chunk structure: some send data: [DONE], others just close the connection; some include a usage block in the final chunk, others don't report usage at all in streaming scenarios and only report it for non-streaming calls. For a gateway developer, the real meaning of "compatible" is "compatible with the handful of paths you happened to test."

Error structure differences are just as thorny. One provider's 400 is JSON with an error.message field, another is a bare string, and a third returns 200 but hides the error in the body. The gateway has to normalize all of this into a form that won't break clients' existing error-handling logic—otherwise the phrase "zero code changes" becomes a lie the first time an upstream sneezes.

Pitfall Two: Usage Metering Is a Schema Problem, Not a Code Problem

This is the part that nearly sank the project.

Billing by token means token counts are money, and money should never be represented with floating point. Unit prices hover between roughly 0.1 and 2 RMB per million tokens, making per-request amounts tiny; if you do arithmetic with float64, errors accumulate and compound across millions of rows. The author's approach: NUMERIC in Postgres, integer micro-units on the hot path, and an explicit decision about which layer rounding happens at.

Reconciliation is even messier. Because providers report usage inconsistently (see the previous point), you can't blindly trust upstream numbers. The final solution was to parse usage on the gateway side, keep a local estimation fallback for streaming requests that never report usage, and retain per-request audit detail so any bill can be traced back to the original event.

The author's lesson is blunt: decide how money is represented before you write the first line of billing code, not after you discover an unexplained 0.3% discrepancy.

Why money can't use floating point: IEEE 754 double-precision floating point (float64) can only exactly represent fractions whose denominator is a power of 2. Decimal fractions like 0.1 and 0.2 are infinite repeating decimals in binary, so they're necessarily truncated when stored. A single error might be on the order of 10⁻¹⁶, seemingly negligible; but when unit prices are as low as 0.1 RMB per million tokens and per-request amounts are themselves tiny, the error's proportion relative to the measured value gets significantly amplified. Multiply that across millions of requests and stack monthly settlement on top, and you get discrepancies that won't reconcile. NUMERIC is PostgreSQL's arbitrary-precision decimal type, storing values exactly as decimal literals, suitable for money fields; the so-called "integer micro-units" approach represents 1 RMB as 1,000,000 minimum units, stored and computed with BIGINT. Both guarantee exactness for addition and subtraction. The key isn't just choosing a type but explicitly deciding which layer rounding happens at—whether at per-request billing time, daily aggregation, or invoice generation—because different rounding points yield different totals.

The role of ClickHouse and Kafka in this chain: Kafka serves as a buffer queue for usage events, ensuring the upstream gateway's write path isn't blocked by jitter in the downstream analytics store; ClickHouse is a columnar OLAP database, good at aggregate scans over massive events (like "total tokens for account X in month Y") but not at high-frequency single-row updates and transactions. So raw usage events go into ClickHouse for analysis and auditing, while billing, model, and provider metadata requiring transactional consistency go into PostgreSQL. The two have different write latencies and consistency semantics, which also explains why the author says "self-hosted ClickHouse is one more thing a two-person team has to babysit at 2 a.m."—it requires independent capacity planning and operations.

Pitfall Three: Hot Reload Matters More Than Redeployment

Provider keys initially required a restart to take effect, meaning every key rotation required a redeployment, which meant a window during which some request was bound to fail.

The fix is unglamorous but eliminates an entire class of operational pain: the gateway polls an internal active providers endpoint every 30 seconds, atomically swapping the in-memory provider set. Keys are encrypted at rest with AES-256-GCM and only decrypted in memory. Rotating a key is now "save it in the admin console, wait 30 seconds, done." The author's advice: if you're building anything that talks to multiple third-party APIs, build the hot reload path before you need it—operational pain grows faster than feature requirements.

AES-256-GCM and encryption at rest: AES-256 refers to a symmetric encryption algorithm with a 256-bit key, brute-force cracking of which is unrealistic under foreseeable compute. GCM (Galois/Counter Mode) is an authenticated encryption mode with associated data that provides integrity verification in addition to confidentiality, detecting whether ciphertext has been tampered with. So-called "encryption at rest" means keys aren't stored in plaintext on disk; the database holds ciphertext, and the gateway decrypts to memory using a master key at startup, existing in plaintext form only briefly in memory. This narrows the exposure surface from "database is read" to "process memory is read," a common starting posture for key management.

Note that the master key itself still needs protection (environment variables, KMS, or systemd's credential mechanism), otherwise encryption at rest merely pushes the problem back a layer. The hot reload path itself must also consider invalidation semantics: the polling interval (30 seconds here) determines the effective latency of key rotation; if a key needs emergency revocation, a faster active invalidation channel is still required.

Pitfall Four: Rate Limiting Needs Two Dimensions, Not One

The author's initial rate limiter had just a single per-key QPS number. It failed in a subtle way: internal sub-requests and user-facing requests consumed the same budget, so a free-tier account with QPS=1 rate-limited itself.

The version that works has two dimensions—a stable per-key rate plus a burst bucket scaled by account age and spend. New accounts get only a small burst allowance, while accounts with history get a more generous one. It's crude, but it kills almost all automated abuse without touching genuine heavy users. The counter operations themselves need care too: application-layer read-then-write creates races under load, and check-and-increment should live in an atomic Lua script against Redis.

Token buckets and atomic operations: The token bucket is a classic rate-limiting model, where the bucket refills tokens at a constant rate, each request consumes some tokens, and requests are rejected when the bucket is empty. Compared to fixed-window counters, it allows short bursts while constraining long-term average rate—which is precisely the theoretical basis for the author's "stable per-key rate + burst bucket scaled by account age and spend."

Why read-then-write creates races: The standard rate-limiting logic is "read current count → check if over limit → increment count." Under concurrency, two requests might read the same old value simultaneously, both judge themselves under the limit, and each increments, causing the actual allowed volume to exceed the threshold. The general solution is to make "check and increment" an indivisible atomic operation. Redis Lua scripts execute single-threaded and uninterruptibly, so writing the judgment logic and data modification together into a script completes the whole action within a single script call, avoiding the window created by application-layer round trips. Similarly, Redis's INCR, SET NX, and other native commands are common atomic building blocks. This also echoes the author's lesson about "internal sub-requests and user requests sharing the same budget"—if the rate-limiting dimensions are designed wrong, no amount of atomicity will save you.

Pitfall Five: The base_url Trap

Small, embarrassing, but worth writing down. Most OpenAI SDKs append paths to your base_url, so if your docs say https://api.example.com and the SDK appends /v1/chat/completions, you get a 404. Some SDKs want the hostname, others want the version with /v1. Get this wrong and your very first integration fails, for a reason that has nothing to do with your actual product.

What the Author Would Do Differently

• Write billing correctness tests first. Not because billing is fun, but because everything else is cheap to change—and it isn't.
• Assume every upstream field is optional. Design the normalization layer around the worst reporter, then optimize for the good ones.
• Don't self-host ClickHouse at this scale. It's excellent and the correct choice on paper, but for a two-person team it's just one more thing to babysit at 2 a.m.
• Ask the distribution question on day one. See below.

Current State: Zero Users, Zero Revenue

The author's status report is honest: it works and has been validated end-to-end (real upstream calls → real model responses → HTTP 200, documented in the repo), but has zero users and zero revenue.

"I built it. I didn't distribute it. Those are two different skills, and I'm only good at one of them"—which is exactly the thing no one tells you when you hear "just build it and users will come."

The code is public at the GitHub repo page6668/He-API. The author is especially willing to answer questions in the comments about usage metering and hot reload, since those are where he'd do things differently the second time around. Additionally, if anyone in Southeast Asia or the Middle East wants to run an API reselling business on this cost advantage, the source code, live environment, domain, ops scripts, and handover docs are all negotiable as a package.

For most developers, the genuinely valuable part of this article isn't the architecture diagram but the ranking of those five pitfalls: billing correctness comes first, and the distribution question comes last, where it's most easily overlooked.

Additional Context

Industry background on "OpenAI compatible": OpenAI's Chat Completions API has effectively become the industry de facto standard, with a large number of SDKs, frameworks, and tools defaulting to it as their interaction convention (e.g., base_url, the /v1/chat/completions path, choices[].delta streaming increments, the usage field structure). So new model providers generally choose "OpenAI compatible" rather than inventing their own protocol, to reuse the entire ecosystem's clients. But since OpenAI doesn't provide a formalized compatibility test suite, implementations diverge noticeably on edge paths: streaming end signals, error body structures, when usage appears under different call modes, finish_reason values, and so on. This is precisely the structural reason behind the original's "the real meaning of compatible is compatible with the handful of paths you happened to test"—compatibility is an approximation of the contract, not the contract itself. For gateway developers, the practical approach is to treat the normalization layer as an independent component, design field optionality around the "worst reporter," and maintain a set of contract tests for each provider.

One-line Summary

Self-hosted API gateway He-API unifies six Chinese models; the real difficulty is token billing and distribution.

Paragraph Summary

He-API is a self-hosted OpenAI-compatible API gateway that unifies six Chinese model providers—DeepSeek, Kimi, Qwen, Doubao, ERNIE, GLM—behind one endpoint and one key, motivated by the fact that Chinese models cost 80%–90% less than Western ones but offer a fragmented integration experience. The author spent two full-time months building a Go gateway, Redis rate limiting, a Kafka + ClickHouse usage pipeline, and PostgreSQL metadata storage. What nearly sank the project wasn't architecture but token billing: money can't use floating point, upstream usage reporting is inconsistent, and reconciliation must be traceable. There are also "compatibility" traps around streaming and error formats, hot reload, two-dimensional rate limiting, and base_url concatenation. The project is validated end-to-end but has zero users and zero revenue, and the author stresses that building and distributing are two different skills.