RADAR: Catching Elusive Gray Failures with Anomaly Detection

What Are Gray Failures?

In site reliability and operations, the most damaging outages are often not the obvious ones — they're the problems your monitoring never flags. As the source material puts it: "Some of the most damaging outages are the ones your monitoring never flags" — a subset of requests slowing down, a replica silently failing, a small slice of traffic quietly being dropped. These are classic examples of gray failures.

What makes gray failures so insidious is that they never trigger traditional threshold-based alerts. Overall system metrics may look perfectly healthy, while localized performance degradation or partial failures continue to impact a subset of users. By the time the problem accumulates enough to cross a conventional monitoring threshold, the damage is already done.

RADAR and the Anomaly Detection Approach

The core idea behind RADAR is to use anomaly detection to fill the blind spots left by traditional monitoring. Rather than relying on fixed thresholds, anomaly detection focuses on how metrics deviate from historical baselines — even when absolute values remain within a "normal" range, unusual behavioral patterns can still be identified.

The value of this approach is that it doesn't require operators to pre-define precise thresholds for every conceivable failure scenario. In distributed systems with vast numbers of metric dimensions and highly variable failure modes, manually configuring thresholds for each one is practically infeasible. Pattern-based anomaly detection automatically adapts to fluctuations in traffic and load, enabling earlier detection of signals that "look fine on the surface but are already broken underneath."

In practice, anomaly detection generally follows a few mainstream technical approaches: statistical model-based methods (e.g., Z-score, moving average deviation) work well when metric distributions are relatively stable; time series decomposition methods (e.g., Facebook's open-source Prophet) separate trend, seasonality, and residuals — well-suited for metrics with clear business peak-and-trough patterns; and machine learning-based methods (e.g., Isolation Forest, LSTM autoencoders) can capture complex cross-metric correlation anomalies. RADAR's core value lies in systematically embedding these techniques into the monitoring pipeline so that detection results feed into downstream processes like alerting and root cause analysis — rather than remaining an offline analysis tool.

Why Traditional Monitoring Misses These Problems

Traditional monitoring systems are built around clear success/failure determinations and static thresholds. When a service node goes completely down, health checks fail immediately and trigger an alert. But when a node merely becomes slow, or only affects 5% of requests, the aggregated average metrics get "diluted" by the large volume of healthy requests and never breach the alert threshold.

This is the fundamental reason gray failures are so hard to catch:

• Locality: The failure affects only a subset of replicas or traffic; global metrics remain insensitive
• Gradual onset: Performance degradation accumulates slowly, with no sharp inflection point
• Silence: Requests are dropped or degraded without the system throwing explicit errors

By establishing multi-dimensional behavioral baselines, anomaly detection stays sensitive to these subtle deviations, enabling early warning before the failure escalates.

Implications for Systems Reliability Engineering

For teams responsible for large-scale system reliability, the direction RADAR represents is practically significant. As system complexity increases, the approach of relying purely on manually crafted alerting rules is hitting a ceiling. Incorporating anomaly detection into the monitoring stack represents a shift from reactively responding to known failures to proactively discovering unknown anomalies.

It's worth noting that anomaly detection is not a silver bullet. It comes with its own engineering challenges — false positive rates, baseline drift, and cold start problems — all of which require tuning for specific business contexts. But as a complement to traditional threshold-based alerting, it offers a viable path toward catching the gray failures that are easiest to overlook and most likely to cause hidden, ongoing damage.

Baseline drift is one of the most common engineering challenges when deploying anomaly detection in production: after a release, traffic growth, or seasonal shift, the system's normal behavior itself changes. If the detection model doesn't update its baseline in time, it leads to persistent false positives or missed detections. Cold start issues arise when new services or new metrics are first introduced, where insufficient historical data prevents a reliable baseline from being established. Common approaches to address both include: sliding-window dynamic baseline updates, triggering a baseline reset after deployment events, and enforcing a minimum observation period before enabling detection. These engineering details determine whether anomaly detection can run stably in production — or whether it remains stuck at the proof-of-concept stage.