3 AM Alert: Building a Disturbance-Free System with Tiered Monitoring and Self-Healing Scripts

Title

3 AM Alert: Building a Disturbance-Free System with Tiered Monitoring and Self-Healing Scripts

Meta Description

Starting from a 3 AM ZFS alert, this article explains three-tier alert classification (L1/L2/L3), a self-hosted monitoring stack, and the "one check, one action" pattern for self-healing scripts — teaching you to reduce alert fatigue through calibration rather than more dashboards, and to build an operations system that doesn't need you awake.

Article Content

At 3:17 AM, my phone buzzed. The screen glared: "ZFS pool degraded — disk 3 checksum errors increasing, scrub recommended." I looked at it for four seconds, put the phone down, and went back to sleep. After coffee the next morning, I ran a scrub; the pool was fine, and the disk went onto a weekly watch list.

The reason this story is boring is precisely because the system was designed to make alerts boring. A year ago, I might not have known something was wrong until the pool entered a severely degraded state — that would have been a completely different 3 AM experience.

The author's core insight is this: most monitoring failures are not a problem with the monitoring tools, but a calibration failure. The system monitors everything and alerts on everything, and the result is that nothing matters. You start ignoring alerts, and then you miss the one that truly matters.

Alert Tiering: One Question Solves 80% of the Noise

Fixing calibration comes down to one question: If I ignore this for 8 hours, what's the worst case? This question can sort almost every alert into three buckets.

L1 — Wake me immediately. Data loss risk (multi-disk failure causing ZFS pool degradation), security intrusion indicators (unusual outbound connections, authentication failures from suspicious IPs), services with real external exposure (scheduled automation pipelines down). These go through ntfy high-priority push and will actually wake me. There are roughly five such alerts in total.

L2 — Just tell me when I'm awake. Degraded but stable: rising checksum errors on a single disk, containers that crashed and auto-restarted successfully, error rates rising but not yet affecting service, worsening disk health trends. Normal-priority push, doesn't wake me, handled with coffee.

L3 — Log only, no alert. Everything else: backup completion, cron execution results, successful health checks, routine scrub results. These go only to logs and never to push notifications.

The first time the author applied this framework, he deleted about 60% of the existing alerts. Response quality went up, alert fatigue went down, and sleep quality improved along with it. That number is worth noting — in most people's alert lists, six out of ten are noise.

Alert Fatigue is a classic failure mode in monitoring: when the volume of alerts exceeds a person's capacity to handle them, the recipient gradually becomes desensitized to all alerts, including the truly critical one. Research shows that in high-pressure settings such as ICUs, alert fatigue significantly lengthens response times. In SRE practice, one metric is the "alert signal-to-noise ratio" — the proportion of actionable alerts among all alerts. Industry experience generally holds that an on-call engineer should not effectively handle more than single-digit alerts per shift; beyond that scale, people start mass-dismissing notifications. The author's use of "worst case if ignored for 8 hours" for tiering is essentially artificially压低 the alert volume and increasing the information density of each alert, thereby reversing the desensitization process.

Self-Hosted Monitoring Stack: Zero Operating Cost

The author's monitoring setup runs entirely on self-hosted infrastructure, with no cloud dependency and no SaaS fees:

• Uptime Kuma handles service availability, checking each service on a schedule. Downtime alerts come with enough context to judge severity without logging in.
• Netdata handles system metrics: CPU, memory, ZFS pool health, and network throughput are visible in real time.
• Scrutiny handles disk health, aggregating SMART data and trending it so degradation is visible before failure occurs.
• ntfy handles push notifications, with each notification containing the alert level, affected service, and a one-line suggested action.

The key to this combination isn't how advanced the tools are, but that alerts carry their own decision context. If an alert requires you to log into a backend to judge whether it's serious, then it is itself a design failure.

ntfy is an open-source, self-hosted push notification service that sends messages to mobile or desktop clients via HTTP PUT/POST, supporting priority (min/low/default/high/urgent), tags, and click-through links. Compared with email or Slack, its advantage is that it can bypass application-layer mute policies to deliver "truly wake-you-up" high-priority pushes, while data never leaves your own server. In the L1/L2/L3 tiering, the author uses ntfy's priority field to distinguish "wake" from "don't wake" at the transport layer, rather than relying on a person to read the notification content and then decide — this reduces the cognitive burden of middle-of-the-night decisions.

Self-Healing Scripts: One Check, One Action

The mindset shift matters more than the tools: don't ask "how do I fix it when it breaks," ask "how do I make it fix itself." The answer to the second question is often twenty lines of bash — check one condition, take one action, write one log entry, escalate on failure. Write it once, and cron runs it forever.

A few scripts actually running on the author's NAS:

ZFS health check (every 6 hours). Read the checksum error count from zpool status; if it exceeds the threshold, immediately trigger a scrub and send an L2 notification. After the scrub completes, record the result and follow up: if errors are cleared, log it; if they persist, escalate to L1. The author hasn't manually initiated a scrub in months.

Container health check (every 5 minutes). Check all Docker containers for unhealthy or exited states; if found, restart with exponential backoff: wait 1 minute, 5 minutes, 15 minutes. After three consecutive failures, escalate to L1 and stop retrying. This avoids waking people up over transient crash loops while not letting real failures slip through.

Ollama inference check (every 10 minutes). Send a minimal test prompt to the inference endpoint; three consecutive failures trigger a container restart, and after the restart, endpoint health verification is required before marking it recovered. Ollama is already load-bearing infrastructure — supporting half the automation stack — and silent failure is unacceptable.

n8n workflow queue check (every 10 minutes). If queue depth exceeds the threshold for more than 10 minutes, restart the n8n worker and log the triggering event. A backlog nobody notices is exactly how automation pipelines quietly rot.

All scripts follow the same structure: check the condition, take only one action, log the timestamp and result, and notify on failure or on the Nth recurrence within a window. The key constraint is "only one action" — scripts that try to be clever and take multiple recovery paths become unpredictable. One check, one action, escalate if unresolved.

The script the author is most grateful exists is a small watchdog that runs every 5 minutes: it checks whether the dashboard node process is alive and responding on the health endpoint. If not, it performs a dependency check, restarts the process, and logs the recovery. Since deployment, it has silently recovered dashboard crashes 11 times — the author noticed none of those 11 in real time, and the service stayed online the whole time. That is the only statistic that matters.

The watchdog for the core management daemon is a different type: it checks every minute but cannot auto-recover — restarting it causes worse problems than the original failure (the author learned this the hard way). So the script doesn't attempt a fix; it only detects and immediately sends an L1. For processes that cannot safely self-heal, fast detection is the entire self-healing strategy. The goal is from failure to alert within two minutes, and in practice it's consistently under 90 seconds.

Exponential Backoff is a retry strategy in which the wait time after each failure grows by a multiple (e.g., 1 minute, 5 minutes, 15 minutes...) rather than retrying at a fixed interval. Its value lies in two things: first, it avoids "crash loops" triggered by transient failures — a service comes up and dies again, the script repeatedly restarts it, consuming resources and creating alert storms; second, it gives dependent systems a recovery window. In the container health check, the author's setting of "escalate to L1 and stop retrying after three consecutive failures" adds circuit-breaker thinking on top of backoff: retries are capped, and beyond the cap the problem is handed to a human rather than infinitely retried to mask a real failure.

The Step Most Monitoring Guides Miss

They teach you how to install Prometheus and Grafana, but they don't tell you that the hardest part isn't instrumentation — it's calibration. Calibration takes a week of seriously observing which alerts you actually act on and which you dismiss without a glance. That work is worth far more than a better dashboard. An uncalibrated alerting system is worse than no alerting at all: with too many alerts, you learn to ignore them, and you end up worse off than with none. Calibrate first, then install dashboards.

Sleep Is the Metric

The author measures his operations quality with one monthly number: how many times this month did I get out of bed for the homelab. It's zero now, and it has been zero for four consecutive months — including the ZFS checksum incident at the start of this article. Every non-zero is data about a system failure point: either an L2 event was misclassified as L1, or a self-healing script didn't trigger, or it was a genuine L1 event that should have been automated away during the day.

The order is clear: build automation first, then tune calibration. Automation without alerts is blind; alerts without automation just wake you up to do what a bash script could do. Combine the two and calibrate correctly, and you get a system that runs itself and tells you only what you must know. Sleep is the metric: four months, and counting.

Background Notes

Watchdog pattern originates in embedded systems: an independent timer is continuously "fed" by the main program; if the main program hangs and fails to feed it on time, the watchdog triggers a reset. In operations, it refers to an independent checker decoupled from the monitored process, specifically detecting silent failures like "the process is still there but no longer responding." The author specifically notes that the watchdog for the core management daemon "cannot auto-recover," which corresponds to a key principle in watchdog design: reset capability should only be granted to components whose post-reset state is deterministic and idempotent. For stateful processes or those responsible for orchestration, blindly restarting can cause cascading failures; in that case the watchdog degrades to a pure detector, returning decision-making authority to humans.

One-line Summary

Use tiered calibration and self-healing scripts to let monitoring systems run quietly unattended.

Paragraph Summary

Starting from a 3:17 AM ZFS alert, the article points out that most monitoring failures are not tooling problems but calibration failures — too many alerts teach people to ignore them. The author proposes a single question, "what's the worst case if I ignore this for 8 hours," to sort alerts into three tiers: L1 wake-up, L2 handle while awake, and L3 log only, and on that basis deletes about 60% of alerts. At the tool level, he builds a zero-operating-cost self-hosted stack with Uptime Kuma, Netdata, Scrutiny, and ntfy, emphasizing that alerts must carry their own level and suggested action. Self-healing scripts follow the constraint of "one check, one action, escalate on failure," and the watchdog has silently recovered dashboard crashes 11 times. Ultimately, "times forced out of bed per month" serves as the operations quality metric, and it has been zero for four consecutive months.