LLM Rate Limits Compared: OpenAI vs Anthropic vs Gemini

OpenAI, Anthropic, and Gemini LLM Rate Limits Compared

When calling large model APIs in production, the HTTP 429 error is almost an "old friend" every engineer encounters. When request volume or token throughput exceeds the concurrency threshold set by the provider, the API refuses service. Single-turn chat scripts rarely trigger limits, but multi-agent pipelines, parallel extraction tasks, and real-time user traffic often max out the default quota instantly.

Based on an in-depth analysis from the Dev.to tech blog, this article systematically compares the rate limiting mechanisms of the three major providers—OpenAI, Anthropic, and Google Gemini—and explores how to build a "zero-429" LLM calling architecture from both the client and server sides.

LLM Rate Limits Across OpenAI, Anthropic, and Gemini

How Modern LLM Rate Limiting Works

Large model rate limits are multi-dimensional throughput constraints enforced at the project or organization level. Unlike traditional REST APIs, which calculate quota only by requests per second, LLM providers assess consumption across multiple dimensions simultaneously:

RPM (Requests Per Minute): The total number of HTTP calls within a rolling 60-second window.
TPM (Tokens Per Minute): The total input and output tokens processed by all concurrent requests within that window.
RPD (Requests Per Day): A macroscopic cap on total daily operations, usually reset at midnight UTC or Pacific Time.
TPD or Spend Limit: A monetary or usage cap within a fixed interval (e.g., a rolling 10 minutes) to prevent runaway billing.

If any dimension hits its ceiling, the request is rejected with HTTP 429—even if other dimensions have plenty of headroom. For example, sending a 150,000-token prompt to an endpoint allowing 200,000 TPM can succeed, but a second 60,000-token concurrent request two seconds later immediately triggers a 429, even though the application has only consumed 2 of 500 RPM.

More critically, providers do not treat per-minute limits as hard batch counters. They implement token bucket or leaky bucket algorithms with short-term rate smoothing. A 60 RPM account roughly equates to 1 request per second; if 15 concurrent calls are made within 1 second, the token bucket's burst capacity is depleted instantly, even if the previous 59 seconds had zero total.

Token bucket and leaky bucket algorithms are key to understanding burst behavior in rate limiting. A token bucket maintains a bucket of capacity C, adds tokens at a constant rate r, and each request consumes a certain number of tokens; when the bucket is full, new tokens overflow and are discarded. The number of tokens accumulated in the bucket is the allowed instantaneous burst—this is why a 60 RPM account can fire 15 concurrent requests in 1 second: 59 seconds of idle time filled the bucket with burst quota. The leaky bucket algorithm is the opposite: requests enter a queue and flow out at a fixed rate, smoothing bursts but not allowing them. Major cloud vendors and LLM providers mostly use token buckets or variants (such as token buckets with rate smoothing), as they guarantee long-term average rates while tolerating reasonable instantaneous bursts. Understanding this mechanism means clients modeling local quotas must also simulate bucket capacity and refill rate, rather than simply amortizing by requests per minute.

Rolling windows are another easily misunderstood concept. Unlike fixed windows (which clear counters at the top of each minute), a rolling 60-second window means that at any moment, the quota judgment looks back over the entire past 60 seconds. Fixed windows allow up to double the burst traffic at window boundaries (the end of the previous window plus the start of the next), while rolling windows eliminate this loophole but make local client prediction more complex, requiring a queue of request timestamps rather than a simple counter. This explains why client token bucket calibration must rely on the remaining quota and reset time returned by the provider, rather than inferring from local counts alone.

Comparison of the Three Providers' Rate Limiting Mechanisms

Understanding the mechanical differences between upstream providers is a prerequisite for writing client interceptors or configuring gateway routing policies. Each provider reports capacity metrics through non-standard response headers, and the way resource exhaustion is assessed also differs.

OpenAI: Merged Token Metering

The OpenAI API measures RPM and merged TPM by model family. When a request is sent, OpenAI checks whether the estimated prompt tokens plus the requested max_tokens fall within the remaining TPM pool. Setting an excessively high max_tokens on a completion request reserves budget upfront, causing a 429 even when the generated content is very short.

OpenAI returns diagnostic headers on every HTTP 200 and 429 response, including x-ratelimit-remaining-requests (remaining requests in the current window as an integer), x-ratelimit-remaining-tokens (remaining tokens before throttling), and x-ratelimit-reset-requests and x-ratelimit-reset-tokens (time until reset, formatted as duration strings like 20ms or 6m0s). A 429 also includes a retry-after header specifying the number of seconds the client must pause.

Anthropic: Separated Input and Output Tokens

The Anthropic Messages API splits token tracking into input tokens per minute (ITPM) and output tokens per minute (OTPM), plus standard RPM. This distinction reflects the asymmetric computational cost of LLM generation: autoregressive token generation is far more compute-intensive than ingesting input context.

Anthropic isolates limits by model family: Claude 3.5 Sonnet requests use different buckets than Haiku or Opus. More critically, in Anthropic's prompt caching mechanism, cache read tokens are not counted against the ITPM quota. For long-conversation agents, establishing a stable cache prefix can significantly reduce rate limit pressure.

Its response header format also differs: anthropic-ratelimit-input-tokens-remaining reports input token headroom, and reset headers use RFC 3339 timestamps (e.g., 2026-10-15T10:05:00Z) rather than duration strings.

Prompt Caching is an optimization mechanism supported by both Anthropic and Google Gemini that directly alleviates rate limits. The principle is: when multiple requests share the same prefix (such as fixed system instructions, few-shot examples, or long document context), the provider persists the KV cache of that prefix, and subsequent requests that hit the cache do not need to recompute that portion, greatly reducing compute cost and latency. Anthropic explicitly states that cache read tokens are not counted against the ITPM quota, meaning a long-conversation agent that maintains a stable cache prefix may consume far less input quota per turn than the raw context length. To hit the cache, the prompt must meet a minimum cacheable length (typically 1024 tokens for Anthropic) and the prefix must be byte-for-byte identical—any tiny wording change invalidates the cache. Therefore, when constructing agent prompts, stable system instructions and static knowledge should be placed first, and dynamic content (user input, retrieval results) last, to maximize cache hit rate.

It is important to distinguish that ITPM (input tokens per minute) and OTPM (output tokens per minute) here are independent quotas, not merged metering. Autoregressive generation requires one forward pass per output token, with a computational cost far higher than parallel input token processing, so providers limit the two separately. This means even with ample input quota, generating large amounts of text can still hit OTPM and be throttled. Clients modeling this must maintain separate token buckets for input and output, and pay special attention to the fact that the max_tokens parameter reserves output quota.

Google Gemini: Project-Level Shared Quota

The Google Gemini API's rate limits operate strictly at the Google Cloud project level, not per API key. This means all background tasks, local development environments, and production microservices share the same quota bucket.

Gemini enforces RPM, TPM, and daily RPD quotas, and paid tiers have a rolling 10-minute spend limit. When quota is exhausted, the API does not return a custom header suite on successful calls; instead, it throws a gRPC-style RESOURCE_EXHAUSTED error wrapped in an HTTP 429 response. Clients must inspect the error response payload or extract the standard retry-after HTTP header to determine the backoff duration.

Why Naive Exponential Backoff Fails in Production

The standard recommendation in client SDK documentation is to wrap requests in an exponential backoff loop with jitter: after a 429, sleep for $t = \min(t_{\max}, 2^{\text{attempt}} \times \text{base}) + \text{jitter}$ for up to three or four retries. This prevents single-threaded command-line scripts from crashing, but under sustained production load it fails for three mathematical reasons.

1. Thundering Herd and Concurrency Amplification

When a highly concurrent cluster (e.g., 50 microservice pods or Celery workers) hits a shared TPM ceiling, all 50 workers receive 429s simultaneously. Even with full random jitter introduced, the cluster remains in lockstep. Once the jitter window passes, 50 retries surge toward the upstream provider simultaneously at the moment the token bucket partially refills, instantly depleting the newly replenished tokens, producing another wave of 429s and doubling the total request queue.

2. Tail Latency Explosion and Upstream Timeouts

Exponential backoff absorbs rate limits by converting quota deficits into client wait time. If the upstream provider needs a 30-second backoff to replenish a 400,000-token prompt quota, downstream HTTP clients, API gateways, or edge load balancers (such as Cloudflare or AWS ALB) will disconnect with a 504 Gateway Timeout before the retry succeeds.

Standard retry algorithms do not calculate token cost. If an application tries to process a 60,000-token context document while the upstream provider's rolling bucket has only 10,000 tokens left, retrying three seconds later is mathematically doomed to fail. What clients need is proactive flow control, not reactive apologies.

Thundering Herd is a classic distributed systems problem, originally used to describe the phenomenon of many processes contending for the same resource at the same moment. In rate limiting scenarios, its trigger chain is: all clients fail simultaneously due to shared quota → they enter backoff simultaneously → they retry at similar times because they use similar or identical backoff formulas → they simultaneously exhaust the just-replenished quota again → they fail simultaneously again. Adding random jitter only mitigates rather than eradicates this problem, because when the number of concurrent clients is large enough, even with a uniform distribution within the jitter window, the number of requests arriving per unit time can still exceed the quota refill rate. The real solution is to base retry decisions on globally shared quota state rather than independent timers—which is precisely why a centralized gateway exists.

Tail Latency refers to high-percentile latency values (such as P99 or P99.9) in the request latency distribution. In a call chain, occasional slow upstream requests amplify layer by layer: a P99 latency of a low-level service can become the P50 latency of the overall request after multiple layers of aggregation. The tail latency problem in rate limiting is that exponential backoff turns a quota shortage that should be resolved quickly upstream into a long client-side hang. When the backoff time approaches or exceeds the timeout threshold of intermediate layers (gateways, load balancers), the connection is cut early by the intermediate layer, so the retry never completes, and the client falls into a "retry-timeout-retry" loop while continuing to hold concurrent connection resources.

Proactive Flow Control differs fundamentally from reactive retry in the direction of control flow. The reactive mode is "send the request first, back off if it fails"; the proactive mode is "confirm sufficient local quota before sending, and queue if not." Proactive mode requires three capabilities: accurate request cost estimation (local tokenization), real-time quota state (local token bucket plus provider response header calibration), and a request scheduler (priority queue and precise timers). This is essentially a port of classic network congestion control to the application layer.

Designing a Client-Side Rate Limiting Engine

To ensure the application client never generates an HTTP 429 status code, engineering teams must invert the control flow: rate limiting must be tracked, budgeted, and scheduled locally before the network packet is sent.

Layered Token Bucket Algorithm

The core engine relies on a thread-safe local token bucket algorithm that tracks requests and tokens simultaneously. The token bucket has a maximum capacity of $C$ and refills continuously at a rate of $r$ tokens per second. Before an outbound request is sent:

The client computes the exact prompt token count using a local tokenizer (such as tiktoken for OpenAI models).
The client checks both the request bucket ($C_{\text{rpm}}, r_{\text{rpm}}$) and the token bucket ($C_{\text{tpm}}, r_{\text{tpm}}$).
If tokens are insufficient, the thread is not dispatched; instead, it is parked in a priority FIFO queue with a precise wake-up timer: $\Delta t = \frac{\text{Tokens Required} - \text{Tokens Available}}{r}$.
After receiving the API response, the client parses the returned rate limit headers and calibrates the local clock with the provider's server-side state.

Compared with passively waiting for a 429 and retrying, this proactive client-side governor knows whether a request can be sent right now before it is sent, fundamentally eliminating 429 errors.

Centralized Rate Limiting at the Infrastructure Layer

A client-side token bucket wrapper works well for standalone processes, but distributed architectures spanning multiple application containers break isolated client-side governors. If 10 container instances run in Kubernetes, each maintaining its own token bucket without inter-process coordination, the entire cluster will frequently exhaust the upstream provider quota.

Moving rate limit enforcement to a centralized infrastructure layer solves this problem. The open-source AI gateway Bifrost intercepts LLM calls as a unified entry point, arbitrates rate limits across multiple keys and providers, and exposes an OpenAI-compatible API as a plug-and-play replacement, requiring applications to change only one base URL. According to its public benchmarks, Bifrost adds only 11 microseconds of overhead under a sustained load of 5,000 requests per second.

Bifrost's core capabilities include:

Virtual keys and budget enforcement: Decouple upstream vendor API credentials from internal application consumers, and enforce hierarchical budget control through sk-bf-* virtual keys (customer → team → virtual key → upstream provider configuration), with independent RPM and TPM thresholds at each layer.
Multi-key load balancing: Automatically weighted routing across API keys. If key A's response headers show remaining tokens below 10%, Bifrost routes subsequent calls to key B within sub-microseconds.
Cross-provider fallback: When an upstream provider returns a 429 or 5xx error, Bifrost retries the request to a preconfigured secondary model or provider. For example, a prompt routed to claude-3-5-sonnet is converted to an OpenAI-compatible format and dispatched to gpt-4o on Azure OpenAI or AWS Bedrock when the primary Anthropic account is limited.
Semantic caching: Store prompts and completions from previous requests and return directly from cache when subsequent requests are highly semantically similar, eliminating both request count and token consumption.

Virtual Key is the core abstraction of enterprise-grade AI gateways. Traditionally, each application directly holds the upstream provider's real API key, which brings three problems: key leakage risk, inability to audit at a fine granularity, and inability to coordinate quota across applications. A virtual key replaces the real key with an internal key issued by the gateway (such as Bifrost's sk-bf-* format); applications communicate only with the gateway, and the gateway injects the real credentials on the egress side. This enables the gateway to enforce multiple governance actions on the same request path: tracking usage by virtual key, enforcing hierarchical budgets (customer → team → virtual key), rotating across multiple upstream keys to avoid single-key limits, and recording complete audit logs. When a virtual key exceeds its limit, it only needs to be throttled or degraded on the gateway side without touching application code.

Multi-key load balancing and cross-provider fallback together form a quota hedging strategy. Since each API key (or each GCP project) has an independent quota, pooling multiple keys is equivalent to adding their total quotas. The gateway dynamically adjusts routing weights by parsing the remaining quota in each key's response headers—keys with low remaining quota are downweighted, and keys with high remaining quota take on more traffic. When all keys or an entire provider return 429, the fallback mechanism converts the request format and forwards it to a backup provider (e.g., from Anthropic to Azure OpenAI). This fallback requires handling differences in API schemas, token metering, and content policies across providers, which is the main source of complexity for self-built solutions and the core value of mature gateways.

Semantic caching goes beyond traditional exact-match caching. It computes request semantic similarity based on embedding vectors and returns cached results directly when a new request's similarity to a historical request exceeds a threshold. Its significance for rate limiting is twofold: it reduces RPM counts and eliminates TPM consumption. However, semantic caching must be applied cautiously—for factual Q&A it may return outdated or approximately incorrect answers, so it is usually only suitable for high-frequency repeated queries, templated tasks, or scenarios with lower precision requirements.

Zero-429 Architecture Checklist

To achieve zero 429 errors in high-throughput enterprise deployments, teams should implement the following architecture checklist:

Compute tokens before transport: Run local BPE tokenization to budget token allocation rather than guessing based on character count.
Never send unbounded max_tokens: Always limit max_tokens to realistic generation bounds to prevent the provider from over-reserving TPM capacity.
Parse provider rate limit headers: Ingest remaining request and token counts on every successful HTTP 200 response to dynamically calibrate local bucket refill rates.
Deploy a centralized AI gateway: Offload multi-key pooling, cross-provider fallback, and shared token tracking to the infrastructure layer.
Implement prompt prefix caching: Construct agent prompts with stable system instructions to ensure cache hits bypass the ITPM quota.
Decouple interactive and background tasks: Isolate batch tasks (data extraction, evaluation pipelines, offline embeddings) onto dedicated API keys or strictly low-priority virtual keys.

FAQ

What is the HTTP 429 status code in LLM APIs?

HTTP 429 indicates that the application has exceeded provider limits on requests per minute, tokens per minute, or daily usage quota. LLM providers return this error to protect shared inference clusters from compute starvation and to enforce commercial account tier thresholds.

Why does exponential backoff fail during LLM traffic spikes?

Standard exponential backoff fails during traffic spikes because retry requests re-enter the queue without token awareness. When multiple concurrent clients hit rate limits simultaneously, backoff causes them to retry in lockstep, instantly depleting newly replenished tokens and triggering cascading 429 errors across the application cluster.

What is the difference between RPM and TPM?

RPM measures the raw count of HTTP API requests initiated within a 60-second window, regardless of payload size. TPM measures the total input and output text tokens processed by those requests. A system can be rate limited by TPM while far below its RPM limit.

Does prompt caching help reduce rate limits?

Yes, prompt caching significantly reduces rate limit pressure on platforms such as Anthropic and Google Gemini. Anthropic does not count cache read tokens against the ITPM limit, allowing applications with stable prompt prefixes to handle higher throughput without triggering 429 errors.

One-line Summary

A systematic comparison of rate limit differences across the three major LLM providers, plus client-side and gateway solutions for a zero-429 architecture.

Paragraph Summary

This article provides an in-depth analysis of the differences in rate limiting mechanisms among OpenAI, Anthropic, and Gemini: OpenAI uses merged token metering, Anthropic separates input and output tokens with cache reads excluded from ITPM, and Gemini shares quota at the project level. It points out that traditional naive exponential backoff fails in production due to the thundering herd effect, tail latency explosion, and lack of token awareness. To address this, the author proposes two approaches: a client-side layered token bucket algorithm for proactive flow control that budgets and schedules quota before requests are sent, and an infrastructure-side centralized AI gateway that governs through virtual keys, multi-key load balancing, cross-provider fallback, and semantic caching. The article concludes with a six-item checklist for a zero-429 architecture.