The Hidden Cost War of LLM Gateways: 5.5% Token Tax vs. Per-Request Logging Fees
Whether an LLM gateway charges a token percentage or per-request fee directly determines AI SaaS profit margins.
Using a scenario of $10K monthly token spend and 5M monthly requests, the article breaks down the real unit economics of five LLM gateways: OpenRouter, Cloudflare AI Gateway, Portkey.ai, Helicone, and LiteLLM Proxy. The core finding is that gateways operate on two distinct billing axes—token top-up percentage tax (5.0%–5.5%) and per-request logging fees (about $9 per 100K requests)—with costs diverging widely across traffic profiles, and a $545/month gap between the cheapest and most expensive options. It distills three rules: drop percentage markups once monthly token spend exceeds $2,000, hit prompt caches via static prefix ordering, and avoid feature flag anonymous contexts and uncached remote evaluations inflating bills.
Why LLM Gateway Pricing Models Determine Your Profit
When most AI SaaS teams connect production traffic to an LLM gateway, they focus on multi-provider automatic failover, unified observability, and prompt caching. But few calculate upfront whether they're paying a percentage of total token spend or a per-request logging fee.
The original article presents a concrete scenario: $10,000 in monthly LLM token spend and 5 million requests per month. Under these parameters, an architectural choice that looks roughly equivalent on paper creates a $545/month ($6,540/year) profit gap.
This isn't theoretical. Below is a breakdown of the real unit economics for five options: OpenRouter, Cloudflare AI Gateway, Portkey.ai, Helicone, and LiteLLM Proxy.
Two Billing Axes: Token Markup Tax vs. Per-Request Logging Fee
The original article splits LLM gateway pricing models into two completely different axes.
Token Top-Up Percentage Tax (5.0%–5.5%)
OpenRouter Standard charges a 5.5% fee on top-ups (minimum $0.80), and charges 5.0% on BYOK traffic after the first 1 million free requests per month. At $10,000 monthly spend, using OpenRouter credits adds $550/month in pure gateway overhead.
Cloudflare AI Gateway pricing depends on the mode. When purchasing credits through Cloudflare unified billing, it charges a 5.0% top-up fee, i.e., $500/month. But in BYOK mode—passing in your own OpenAI, Anthropic, or Gemini keys—the token markup is 0%, and the cost is just the $5/month Workers Paid base plan.
BYOK (Bring Your Own Key) means enterprises use API keys obtained through their own direct contracts with model providers (OpenAI, Anthropic, Google, etc.) and forward requests through the gateway. In this case, the gateway does not participate in token resale, so it should theoretically not charge a token markup; its revenue can only come from infrastructure subscription fees or per-request billing. This is why Cloudflare AI Gateway has a 0% markup in BYOK mode, while purchasing through the platform's unified credits incurs a 5% top-up fee—the latter is essentially the margin the gateway earns as a token reseller. For enterprises, BYOK also means bills go directly into their own cloud contracts, allowing them to enjoy the provider's tiered discounts and credits, but they need to manage key rotation and quotas themselves—an operational cost that is easily overlooked during selection.
The Per-Request Logging Trap (About $9 per 100K Requests)
Portkey.ai does not charge a token markup (100% BYOK), but bills by request log volume: a $49/month base fee plus about $9 per 100K logged requests. If an application makes 5 million lightweight classification or routing calls per month, the Portkey bill reaches $441/month—even if the underlying token spend is small.
Helicone Pro also has a 0% token markup, with a $79/month base fee (including 4 seats) plus tiered log storage (about $60/month for 5 million requests), totaling about $139/month.
LiteLLM Proxy is an open-source self-hosted solution that runs as a stateless Docker container on Railway or Fly.io with its own Postgres database, costing about $12/month, with 0% token markup and unlimited request routing.
Full Monthly Cost Comparison Across Six Options
Based on $10,000 monthly token spend and 5 million requests per month, the original article provides a complete comparison table.
| Gateway Option | Billing Model | Token Markup | BYOK Policy | Base + Logging Fee (5M Requests) | Total Monthly Cost |
|---|---|---|---|---|---|
| Cloudflare AI Gateway (BYOK) | Workers Paid base plan | 0.0% | 100% BYOK (free) | $5 | $5 |
| LiteLLM Proxy (self-hosted OSS) | Self-hosted Docker | 0.0% | 100% BYOK (free) | ~$12 | $12 |
| Helicone Pro | Base + tiered log storage | 0.0% | 100% BYOK | $79 + ~$60 | $139 |
| Portkey.ai Growth | Base + per 100K request logs | 0.0% | 100% BYOK | $49 + ~$392 | $441 |
| Cloudflare AI Gateway (unified billing) | 5.0% top-up fee | 5.0% | Uses unified credits | $0 | $500 |
| OpenRouter Standard | 5.5% top-up fee | 5.5% | 5% after 1M requests | $0 | $550 |
The difference between the cheapest and most expensive options is $545/month, even though they are nearly indistinguishable on feature lists.
Three LLM Routing and Cost Control Rules
The original article distills three actionable rules from this data.
Don't Pay a Percentage Token Markup Once Monthly Spend Exceeds $2,000
Above this threshold, percentage billing quickly outpaces fixed infrastructure costs. The solution is straightforward: use Cloudflare AI Gateway's BYOK mode ($5/month) or self-hosted LiteLLM Proxy ($12/month), turning multi-provider failover routing into a fixed infrastructure expense rather than skimming 5% to 5.5% off gross AI revenue.
Align Static System Prefixes to Hit Native Prompt Caching
Anthropic's prompt caching (1.25x cost for 5-minute writes, then reads at up to a 90% discount) and OpenAI's automatic prompt caching (50%–75% off matched prefixes over 1024 tokens) both require deterministic prefix ordering. The approach is to put static tool definitions and system prompts first, and append dynamic user/RAG context to the very end of the message array. If the order is disrupted, cache misses occur and the discount is lost.
Prompt Caching is a provider-side optimization: when the prefix of a request exactly matches a recent request, the provider reuses the already-computed KV cache, skipping redundant attention computation. Anthropic uses explicit caching, requiring cache_control breakpoints to be marked in the request; writes are billed at 1.25x, cache hits read at about a 90% discount, and the default lifetime is 5 minutes. OpenAI uses automatic prefix caching, where matching prefixes over 1024 tokens automatically receive a discount without explicit declaration. The hard prerequisite shared by both is prefix determinism—any dynamic content in the prefix (timestamps, user IDs, randomly ordered tool lists) changes the cache key and drives the hit rate to zero. Therefore, cache optimization is essentially a structural problem of prompt engineering, not merely a gateway configuration issue.
Beware the Feature Flag Evaluation and Anonymous Context Cliff
When using feature flags for model experiments (gpt-4o-mini vs. claude-3-7-sonnet), don't poll a remote flag API on every unauthenticated page load. The original article provides the numbers: on LaunchDarkly, anonymous visitors drive up client-side monthly active context costs (about $1,020/month at 100K MAU); on Statsig or PostHog, uncached remote flag checks inflate impression-based bills. The recommendation is to use server-side local evaluation mode (GrowthBook, Flagsmith, or PostHog local evaluation with caching).
Feature Flag platforms typically bill by "context," meaning each evaluated user or device identifier counts as a metered unit. The problem with anonymous visitors is that logged-out users have no stable ID, so platforms often generate a new context for each session or even each page load, severely inflating MAU counts and bloating bills. Local Evaluation mode uses a server-side SDK to periodically pull the full rule set and evaluate in memory, communicating with the platform only when rules change—converting the billing unit from "per evaluation" to "per rule sync," while also reducing evaluation latency and avoiding flag evaluation failures caused by network jitter.
How to Use These Numbers
The original article includes downloadable CSV/JSON datasets and a zero-dependency Node.js CLI simulator so you can model based on your own token spend, request volume, prompt cache hit rate, and feature flag evaluation costs. However, these datasets and tools are paid products ($24 and $19 Payhip datasets, $49 Gumroad full package), and the article itself is an engineering analysis with a product promotion angle.
Setting aside the commercial promotion, the core conclusion deserves independent consideration: an LLM gateway's pricing structure is more worth carefully calculating during architecture selection than its feature list. A 5.5% token markup is just $55 at $1,000 monthly spend, but it becomes $5,500 at $100,000 monthly spend—at which point the fixed-cost advantage of self-hosting or BYOK mode becomes very obvious. Conversely, per-request billing quietly eats into profits in scenarios with low token spend but high request volume (such as lightweight classification, routing, and embedding calls).
Before choosing a gateway, first calculate which billing axis your traffic structure falls on.
Related articles
LLaMA-Factory in Practice: A Complete No-Code Guide to Fine-Tuning Large Models
An introduction to LLaMA-Factory, the open-source fine-tuning tool: its 60.1K-star background, supported models and algorithms, local vs. cloud deployment, and CUDA setup steps.
Zeng Ming's Deep Retrospective: Why OpenAI May Not Be the Ultimate Winner — and Why the Corporate System Will Disappear
Zeng Ming's deep retrospective on the AI endgame: model companies are just infrastructure, OpenAI and Anthropic won't be the final winners; the next battle is the Agent entry point standard; the corporate system will disappear as AI restructures collaboration, and only mission-driven companies can survive the cycle.
OpenHands v1.28.0: OpenCode and PI Added as Built-in Agents, API Keys Requestable In-Conversation
OpenHands ships 5 releases in 21 days: v1.27.0 adds OpenCode and PI as built-in agents, v1.28.0 enables in-chat key requests. ACP integration, credential reuse, model list changes, and upgrade advice.