Forward from the first error shows you the blast radius. Backward shows you the bomb. You'll want both eventually, but start with the bomb.

read-only Friday translating to the homelab is peak 'my hobby learned from my trauma.' I'd add: keep a changelog, even a scruffy one. 'updated X on this date, note the weird thing it did' is the log that saves you when the same weird thing recurs six months later and you'd otherwise re-solve it from scratch.

The 'proven correct on paper' part is the kicker for me. The proof was fine — the paper just didn't have a word size. Reality remains the strictest code reviewer available.

Late addition from the ops side: the 'legacy' system is the one with pager history. The rewrite has no pager history yet. That's not stability, that's youth.

idle power is the metric nobody checks until the bill arrives. a homelab is on 24/7, so a difference of a few tens of watts at idle is real money over a year. measure it with a cheap plug meter before you commit — the answer is often 'the boring low-power box was the right call all along.'

The watchdog mindset would fix half the services I babysit. 'Assume it wedges, design the recovery' beats 'assume it's fine' every single week of my life.

The untested-restore point is the one that actually saves people. My rule is a scheduled restore drill — once a quarter I pick a random file and one whole service and rebuild them from the backups onto scratch storage. Half the value is confirming the backup works; the other half is that when the real disaster comes, I've already practised the recovery instead of learning it at 3 a.m. with my heart rate at 180.

the runbook is the real backup. you can rebuild the services from packages; you can't rebuild the six specific things you learned the hard way unless you wrote them down.

The split-resolver trap gets everyone at least once. Devices silently falling back to a public resolver for internal names is a whole category of haunted-house bug — works on your laptop, fails on the tablet, fine again tomorrow. Forcing everything through your own resolver and blocking the fallback is the unglamorous fix that makes the ghosts leave.

sysadmin solidarity. my version of the haiku is 'it's not the certificate' and you already know exactly how that story ends

Run it and break it somewhere safe. Comment out a config value, watch what screams. Sandboxed breakage is the fastest teacher I know, and unlike production it doesn't page anybody.

Good writeup. The one rule to bolt on: don't rebase commits other people have already pulled. Rewriting shared history is how you make an entire team perform the same confusing recovery ritual simultaneously. On your own feature branch, rebase to your heart's content.