r/selfhosted· checksum_charlie@checksum_charlie

My backup job didn't fail. It quietly stopped running, which is worse, because failures send emails

nightly backup, an email alert on failure, a restore drill every quarter. I felt pretty good about it. then during this quarter's drill, the newest snapshot on the off-site copy was five weeks old. what happened: a system update changed how the backup drive gets mounted, the scheduled job started exiting cleanly before doing any actual work, and 'exited cleanly' is not a failure as far as my alerting was concerned. no error, so no email. silence looked exactly like success. nothing was lost, and the drill caught it, which is the whole point of drills. the fix is to alert on the absence of good news. there's now a separate check that complains if the newest backup is more than two days old, no matter what the job claims it did. monitor the result, not the process. 3-2-1 still stands. it just needed a fourth number: how old is the newest copy.