Cache Stampede: How to Handle 50,000 Requests Penetrating at Once

The Root Problem: Cache Penetration and the Thundering Herd

Imagine this scenario: a popular livestream or breaking news page where 50,000 viewers request the same cached file in the same millisecond. And at that exact moment, the cache entry has just expired. The result is catastrophic — all 50,000 requests "penetrate" the cache layer and flood the backend database or origin server simultaneously.

This is the classic Cache Stampede problem in distributed systems, also known as the Thundering Herd effect or Dog-piling. Cache Stampede was first systematically studied in large-scale web service architectures. Facebook described this phenomenon and its mitigation strategies in detail in their 2013 paper Scaling Memcache at Facebook. The term "Thundering Herd" originates from the operating systems domain — when multiple processes or threads are waiting on the same event (such as an accept() call), the event trigger wakes all waiters, but only one can successfully handle it. The rest wake up only to immediately go back to sleep, causing massive wasted context switches and resource consumption. Linux kernel versions after 2.6 partially addressed the OS-level thundering herd problem through the WQ_FLAG_EXCLUSIVE flag, but at the application layer in caching scenarios, this challenge still requires dedicated engineering design.

When a hot key expires under high concurrency, massive requests instantly punch through the cache. Backend services are often overwhelmed within milliseconds, triggering a cascading service avalanche.

This discussion from the Reddit tech community touches on a core challenge that every engineer building high-concurrency systems must face. It seems simple on the surface, but it truly tests your comprehensive understanding of caching mechanisms, lock contention, and system degradation.

Why Simply Extending the Expiration Time Doesn't Work

A junior engineer's first instinct is often: "Why not just set a longer cache expiration time?" But this treats the symptom, not the cause.

No matter how long you set the cache TTL, as long as there's an expiration moment, there will always be a possibility of that exact moment being hit by high concurrency. Moreover, an excessively long TTL leads to stale data, sacrificing cache freshness. A real solution needs to address both concurrency control and invalidation strategy.

Three Mainstream Solutions for Cache Stampede

Solution 1: Mutex / Single-flight

The core idea: when a cache entry expires, only one request is allowed to rebuild the cache; all others wait or receive a stale value.

In practice, the first request that discovers a cache miss attempts to acquire a distributed lock (e.g., using Redis's SETNX). The request that successfully acquires the lock is responsible for querying the database and repopulating the cache; the remaining 49,999 requests either block briefly while waiting or immediately return the previous slightly stale cached data.

SETNX is an atomic operation command provided by Redis, short for SET if Not eXists — it only sets the value for a key if that key doesn't already exist, returning success. This property makes it naturally suited for implementing distributed locks. However, in production environments, a simple SETNX carries the risk of the lock never being released (e.g., if the lock-holding process crashes). Therefore, the Redis team officially recommends using the SET key value NX PX milliseconds combined command, which merges locking and timeout setting into a single atomic operation. For higher reliability requirements, Redis creator Antirez proposed the Redlock algorithm, which acquires locks across multiple independent Redis instances simultaneously to avoid single points of failure. However, this algorithm also drew famous criticism from distributed systems expert Martin Kleppmann, who highlighted safety concerns under network partition and clock drift scenarios.

Go's singleflight package is a classic implementation of this pattern — it merges duplicate calls for the same key at the same moment into a single actual execution, with all other callers sharing the same result. This approach can compress 50,000 database requests down to just 1.

singleflight resides in the Go extension library golang.org/x/sync/singleflight. Its core data structure is a map indexed by request key, where each entry contains a sync.WaitGroup and result storage. When the first request arrives, singleflight creates a new entry and executes the actual function call; subsequent requests with the same key discover the existing entry and block via WaitGroup.Wait(). Once the first request completes, all waiters share the same return value. This design is widely used in microservice gateways and API aggregation layers. It's worth noting that singleflight provides process-level deduplication — in multi-instance deployment scenarios, it still needs to be combined with distributed locks to achieve cluster-level request merging.

Solution 2: Logical Expiration

This is a more advanced approach where cache entries never physically expire. Instead, a "logical expiration time" field is stored within the value itself.

When a request detects that the logical time has expired:

• It immediately returns the current stale data (ensuring response speed)
• It simultaneously triggers a background thread to refresh the cache asynchronously

This way, users never experience blocking caused by a cache miss, and the backend only needs to handle the pressure of a single async refresh. The trade-off is that slightly stale data may be returned for a short period, which requires the business to tolerate this eventual consistency.

The design philosophy of logical expiration is closely aligned with the Eventual Consistency model in distributed systems. Under the constraints of the CAP theorem, highly available systems often need to make trade-offs between consistency and availability. Logical expiration prioritizes availability — even if the returned data is a few seconds or even tens of seconds stale, it prevents users from experiencing blocking or errors. This pattern is known as stale-while-revalidate in content delivery networks (CDNs). The HTTP standard (RFC 5861) specifically defines the Cache-Control: stale-while-revalidate directive to support this behavior: a cache can still be used after expiration while asynchronously revalidating with the origin server in the background. Mainstream reverse proxies and CDN edge nodes like Nginx and Varnish natively support this mechanism, making logical expiration not just an application-level design pattern but a mature solution incorporated into internet standards.

Solution 3: Randomized Expiration Times (TTL Jitter)

To address the cache avalanche caused by a large batch of keys expiring at the same moment, a simple yet effective technique is to add a random jitter value to each key's TTL.

For example, instead of uniformly setting 3600 seconds, use 3600 + random(0, 300) seconds. This way, even caches written in the same batch will expire at different times, avoiding a concentrated burst of invalidations.

The effectiveness of TTL Jitter can be understood from a probability theory perspective. Suppose N keys expire simultaneously without jitter, creating N concurrent requests hitting the backend instantaneously. After adding uniformly distributed random jitter [0, J], the expiration times are spread across a window of J seconds, reducing the expected number of keys expiring in any given second to N/J. This concept is widely applied in distributed systems: TCP's exponential backoff retransmission adds randomization to avoid network congestion synchronization (known as jittered exponential backoff), and the AWS official architecture guide lists it as a best practice for avoiding retry storms. It's worth noting that the choice of jitter range J requires a trade-off: too small and the spreading effect is limited; too large and some data may become excessively stale.

The Optimal Solution in Production: Combined Strategies

In real production environments, mature teams rarely rely on a single solution. Instead, they combine them in layers:

• Multi-level cache architecture: Local cache (e.g., Caffeine) + distributed cache (Redis). Local caches absorb the vast majority of duplicate requests from the same node.

Caffeine is the highest-performance local cache library in the Java ecosystem, developed by Ben Manes, the original author of Google Guava Cache. It employs a Window TinyLfu admission policy, an adaptive cache eviction algorithm that combines the strengths of both LRU and LFU, handling both burst traffic (recency) and steady-state hotspots (frequency). In a multi-level cache architecture, L1 local cache (e.g., Caffeine) hit latency is typically at the nanosecond level (~50-100ns), while L2 distributed cache (e.g., Redis) latency is at the millisecond level (0.5-2ms) — a difference of up to 10,000x. Therefore, even with only an 80% hit rate, local caching can reduce Redis QPS pressure by 80%, providing a significant multiplier effect in preventing cache penetration. The main challenge of multi-level caching is consistency maintenance, typically handled through pub-sub mechanisms (e.g., Redis Pub/Sub) to broadcast invalidation notifications across nodes.

• Mutex rebuild + logical expiration: Use the singleflight mechanism to prevent penetration, and logical expiration to avoid blocking.
• Rate limiting and circuit breaking: As the last line of defense — even if cache strategies fail, the backend is protected from being overwhelmed.

Rate Limiting and Circuit Breaking are core protection mechanisms in microservice architectures. Common rate limiting algorithms include Token Bucket and Sliding Window: the former allows a certain degree of burst traffic, while the latter provides more precise rate control. The Circuit Breaker pattern was introduced by Michael Nygard in the book Release It! and was popularized by Netflix's Hystrix library. Subsequent frameworks like Resilience4j and Sentinel have carried on this philosophy. A circuit breaker maintains three states: Closed (normal pass-through), Open (fast fail), and Half-Open (tentative recovery). In a cache avalanche scenario, a circuit breaker can automatically cut off traffic when it detects a spike in backend error rates, buying time for system recovery and preventing cascading failures from spreading across the entire service chain.

• Cache warming and proactive refresh: For known hot data, proactively refresh before expiration to eliminate the invalidation window at the source.

Back to the Original Question: How to Choose the Right Solution

The cleverness of the question "50,000 viewers miss the same cache file in the same millisecond" lies in how it distills concurrency pressure to its extreme. It reminds us: any invalidation mechanism with a time window will be ruthlessly amplified under sufficiently high concurrency.

Great system design doesn't eliminate this window — it uses lock merging, async refresh, and random jitter to reduce the window's impact on the backend to a manageable level. For content delivery scenarios (such as livestreaming or CDN edge nodes), logical expiration with async refresh typically provides the smoothest user experience. For strong data consistency scenarios, the mutex lock approach is preferred.

Understanding these trade-offs is exactly what separates "knowing how to use caching" from "truly understanding caching."

Key Takeaways