restic Backup to B2 Failure Alert Reports Only a Line Number? Reflections on Alert Design

A Backup Failure Alert That Only Gives a Line Number

At 6:40 AM on October 11, 2026, a critical-severity alert fired: the daily restic remote backup job to B2 had failed. The alert text offered only one detail—"exit at line 92."

The job is driven by a systemd timer, which invokes /opt/orqestra/scripts/backup-db.sh via orqestra-backup.timer every day at 5:30. The alert pointed to two directions for investigation: the journal logs of the orqestra-backup unit, and that script on the VPS.

restic is an open-source command-line backup tool written in Go that uses a content-addressed storage model: each backup uploads only the data blocks that don't already exist, so it naturally supports incremental backups and deduplication. It supports a variety of backends, including local directories, SFTP, and S3-compatible object storage—B2 (Backblaze B2) being a common one. When restic runs, it writes progress, warnings, and error messages to stderr, and returns a non-zero exit code on failure. Precisely because it writes the real cause (repository lock conflicts, authentication failures, network interruptions, storage quota overruns, etc.) to its own output, if the script layer doesn't capture that output, whoever debugs the issue can only see the outer script's line number.

The Line Number Tells You "Where It Stopped," Not "Why It Stopped"

The most important piece of information is missing from the alert: what restic itself printed. Was the repository on B2 unreachable? Did restic itself return an error? Or did something go wrong in an earlier step of the script? None of this can be determined.

The line number can only pinpoint where the script was interrupted; it cannot explain why the interruption happened. For troubleshooting, that's almost useless information—the on-call engineer still has to open the logs and start searching from the beginning after receiving the alert. The alert hasn't saved them any time.

A Common Pitfall in Alert Design: Using Status Words and Locations Instead of the Actual Error

Many automation scripts' alerting logic stops at "if the process exit code is non-zero, fire an alert," and then stuffs a line number or status word into the notification. On the surface the information looks complete, but in reality the most critical context—the tool's own error output—is left behind in the logs.

When the job chain is "systemd timer → shell script → restic → B2," the failure can occur at any link in the chain. Reporting only a line number effectively shifts the entire cost of localization onto the person receiving the alert. This is especially costly in scenarios where the alert fires in the middle of the night and a quick judgment is needed on whether manual intervention is required.

systemd timer is the scheduling mechanism provided by systemd, consisting of a pair of units: the .timer unit defines when to trigger (similar to cron), and the .service unit defines the command to actually execute. Compared with cron, its advantages are that you can use systemctl list-timers to see the next trigger time, it supports Persistent=true to catch up on missed jobs, and all output goes uniformly into the journal. You can search the complete logs by unit with journalctl -u orqestra-backup. Understanding this is key: the timer is only responsible for "when to start," and the original evidence of the failure cause always remains in the journal of the corresponding service unit. If the alert text doesn't include that output, it effectively severs the evidence from the notification.

An Improvement Principle You Can Put Into Practice Directly

The original article's advice is very clear: when a restic job runs under a systemd timer via a script, make the alert carry restic's own error output, because a line number plus a status word is not enough to support any action.

Concretely, you can capture the stderr of the restic command in the script and, on failure, write it into the alert body; at the same time, keep the complete original output in the journal as a fallback. That way the alert itself can answer "which step failed and why," rather than directing the engineer to the logs for a second round of searching.

This incident did not identify the root cause, and the author explicitly refused to guess the cause from the line number. Its value doesn't lie in why a particular backup failed, but in a reminder about the information density of alerts: an alert's job is to convey actionable information, not to convey a pointer to the logs.

A common way to capture a subcommand's error output in a shell script is: use err=$(restic backup ... 2>&1 >/dev/null) to collect stderr into a variable, then check $?; or use set -o pipefail together with tee to both write to disk and forward. Note that some of restic's output (such as progress) goes to stderr by default, so it's best to use --quiet or --json to control the output volume, then truncate the captured content (e.g., keep only the last 20 lines) to avoid the alert body being too long and getting collapsed by the notification channel. Embedding this truncated text in the alert lets you judge whether immediate manual intervention is needed without opening the journal.

One-line Summary

A backup failure alert reported only a line number and no error output, a warning that alerts should convey actionable information.

Paragraph Summary

The article recounts a postmortem of a failed restic remote backup alert: a backup script driven by a systemd timer triggered a critical alert on October 11, 2026, but the notification contained only "exit at line 92," giving no indication of restic's own error and making it impossible to tell whether the B2 repository was unreachable, restic returned an error, or an earlier step in the script went wrong. The author points out that a line number can only pinpoint where the script was interrupted, not why, so the on-call engineer still has to dig through the journal from scratch—the alert saved no time at all. This "fire an alert on non-zero exit code, then stuff in a line number or status word" design shifts the entire cost of localization onto the recipient, and is very costly in scenarios where a quick judgment about manual intervention is needed in the middle of the night. The improvement principle is to make the alert carry restic's stderr error output while keeping the complete journal logs as a fallback.