packet_loss@packet_lossIt's always DNS. A homelab horror story with a boring, correct ending
Spent an entire evening convinced my reverse proxy was broken. Services unreachable by name, reachable by IP, intermittent in a way that made no sense. Restarted everything. Read the proxy logs until my eyes crossed. Suspected the certs, the firewall, gremlins. It was DNS. It is always DNS. My internal resolver had a stale record cached from before I renumbered a container, my devices were split between the internal resolver and a public one depending on the day's DHCP mood, and half my lookups were resolving to an address that no longer existed. Everything downstream looked broken because the names were lying. The fix, and the lesson: pick one authoritative internal resolver, point everything at it deliberately, keep your local DNS records in one place you actually maintain, and lower your TTLs while you're actively changing things so stale entries expire fast. Since I did that, my 'the network is broken' incidents dropped to near zero, because they were never the network. They were the phone book.
Join the conversation
Facet is free to read. To reply you need an account: one private root identity, and up to ten public personas that can never be linked to each other or to you.
Create an account
greybeard_junior@greybeard_junior· 7/5/2026, 11:32:20 PMThe split-resolver trap gets everyone at least once. Devices silently falling back to a public resolver for internal names is a whole category of haunted-house bug — works on your laptop, fails on the tablet, fine again tomorrow. Forcing everything through your own resolver and blocking the fallback is the unglamorous fix that makes the ghosts leave.
kernel_of_truth@kernel_of_truth· 7/6/2026, 3:14:40 AMthe TTL tip is the one I always forget in the moment. cranking TTLs way down before a migration and back up after is the difference between changes propagating in seconds versus you staring at a cached lie for an hour swearing at perfectly healthy hardware.
packet_loss@packet_loss· 7/6/2026, 4:52:08 AMand then the classic capstone: you fix it, it still fails, you rage for ten more minutes, and it was your OS's own DNS cache holding the old record. flush the local cache too or the phone book lies from inside the house.
read_only_friday@read_only_friday· 7/6/2026, 5:40:50 AMevery ops veteran has the exact moment in this post bookmarked in their soul. 'reachable by IP, not by name' is DNS's signature confession and it still takes us an hour every time because we refuse to believe it's that again.
checksum_charlie@checksum_charlie· 7/10/2026, 4:40:43 PMcoming back to the DNS thread because it happened to me AGAIN this week, exactly as foretold. renumbered a service, forgot the local record, spent twenty minutes blaming the firewall. it is always DNS, and knowing it's always DNS does not stop it from being DNS. printed the checklist and taped it to the rack this time.