How Uber Defends Against Retry Storms: Fault-Tolerant Design in Distributed Systems

What Is a Retry Storm

In large-scale distributed systems, retry mechanisms are a common technique for improving reliability. When a request fails, the client automatically resends it, expecting a successful response once the transient fault has cleared. This works well in most scenarios — but when the system itself is overloaded or partially degraded, retries can become the last straw that breaks the system entirely.

A retry storm is exactly this kind of cascading reaction: when a downstream service experiences latency or failures, a flood of upstream requests time out and trigger retries. That retry traffic piles on top of an already strained load, causing the downstream service to deteriorate further, which in turn triggers even more timeouts and retries — forming a positive feedback loop that keeps amplifying. For a platform like Uber handling enormous request volumes every second, this amplification effect can turn a localized failure into a global service outage.

Why Retries Spiral Out of Control

The key to understanding retry storms lies in recognizing the "amplification effect" of retry traffic. Suppose every service call is configured with up to 3 retries. When a downstream service starts returning errors, the actual request volume hitting that downstream can instantly balloon to 3–4 times its normal level. In a multi-tier call chain, this amplification compounds at every layer — upstream retries amplify load on the middle tier, middle-tier retries amplify it further downstream, and the pressure on the bottom-most services grows exponentially.

The synchronization problem makes things worse. When large numbers of clients fail simultaneously due to the same downstream fault and retry at the same time, retry requests become tightly clustered in time, creating periodic traffic spikes. Even if the average load isn't high, these spikes can punch through a service's processing capacity in a matter of seconds, making recovery significantly harder.

Uber's Defense Strategies

To address this challenge, Uber employs a multi-layered defense approach. The core idea is to preserve the reliability benefits of retries while suppressing their amplification effect.

Retry Budgets and Rate Limiting

The most direct approach is to set a "retry budget." Rather than allowing unlimited retries, the system caps retry traffic as a percentage of total traffic — for example, allowing only 10% additional retry requests. Once retries exceed this threshold, subsequent failed requests are returned as errors immediately rather than triggering another retry attempt. Even if a downstream service experiences widespread failure, the extra load from retries is kept within a controllable range, preventing exponential amplification.

Circuit Breaker Pattern

The circuit breaker is another critical line of defense. When the error rate of a downstream service persistently exceeds a configured threshold, the circuit breaker "trips" and directly rejects calls to that service for a period of time, rather than letting requests continue to pile up. This gives the downstream service time to breathe and recover, and prevents the upstream from wasting resources on requests that are guaranteed to fail. Circuit breakers are typically used alongside a half-open state, periodically allowing a small number of probe requests through to determine whether the downstream has recovered.

The circuit breaker pattern originates from the fuse concept in electrical engineering and was formally introduced to software by Michael Nygard in Release It!. It typically maintains three states: Closed — requests pass through normally while failure rates are tracked; Open — once the error rate exceeds the threshold, the breaker trips and all requests fail immediately without attempting to reach the downstream; Half-Open — after a cooldown period, a limited number of probe requests are allowed through, and if they succeed, the breaker returns to the Closed state; if they fail, it reopens. The elegance of this state machine is its adaptive awareness of downstream health: it doesn't keep blocking requests after the downstream has recovered, nor does it recklessly allow full traffic when the downstream is still fragile. In practice, threshold calibration requires careful tuning against business tolerance and service SLAs — thresholds set too low cause false trips, while thresholds set too high provide no meaningful protection.

Exponential Backoff with Jitter

To address the traffic spikes caused by synchronized retries, exponential backoff combined with random jitter has become the standard approach. Exponential backoff progressively increases the wait time between retry attempts, avoiding rapid successive hits on a downstream service. Random jitter adds a randomized offset on top of the backoff interval, spreading retry requests that would otherwise cluster at the same moment across a time window — effectively smoothing out the traffic spikes.

On the topic of jitter strategies, the AWS engineering blog conducted a systematic comparison of several common implementations: pure exponential backoff (no jitter), full jitter (wait time fully randomized between 0 and the cap), equal jitter (fixed step plus random offset), and "decorrelated jitter" (each wait time randomized based on the previous actual wait time). The results showed that Full Jitter achieved the best balance between throughput and server-side pressure — it sacrifices the retry latency of individual clients, but dramatically reduces temporal synchronization across the client population, most effectively flattening traffic spikes. The practical implication is counterintuitive: "backing off more" doesn't automatically mean "safer." The degree of randomization in the jitter is the key variable for dispersing synchronized retries.

Implications for Engineering Practice

Uber's approach reveals an important principle: retries are a double-edged sword. When designing resilient systems, it's not enough to think about "how to retry after a failure" — you also need to think about "when to stop retrying." A naive retry strategy may be harmless in small-scale systems, but in high-concurrency, multi-tier architectures, it can become the very mechanism that amplifies failures.

For teams building distributed systems, practical takeaways include: introduce retry budgets rather than fixed retry counts; deploy circuit breakers on both the client and server sides; always pair retry logic with backoff and jitter; and establish observability to monitor retry traffic ratios in real time, catching abnormal amplification early. These mechanisms work best in combination — together they allow you to maintain reliability while holding the line on system stability.

(Note: This article is based on a topic shared on Hacker News. Since the original discussion contains limited detail, specific technical implementations should be verified against Uber's official engineering blog.)