DNS Disasters: How One Ancient Doc and an Overzealous User Took Down an Office Network
If you’ve ever worked in IT, you know there are two immutable laws of the universe: someone will always try to “help,” and it’s (almost) always DNS. Today’s story, courtesy of u/Nstraclassic on r/TalesFromTechSupport, perfectly illustrates both. It’s a wild ride through a network meltdown caused by outdated documentation, a rookie technician, and an end user with just enough admin privileges to cause maximum chaos. Grab your popcorn—and maybe a stress ball.
The Calm Before the DNS Storm
Our tale begins innocently enough. A new, eager L1 tech joins the team, soaking up tickets and knowledge like a sponge. Meanwhile, the seasoned OP is knee-deep in a Domain Controller (DC) migration, with a final cutover scheduled at noon. All systems are (supposedly) go.
Then, as fate (and networks) would have it, a few users report connection issues. The greenhorn L1 jumps into action, asking thoughtful questions about DNS and DHCP. So far, so good. But after an hour of troubleshooting—and a couple of checks from OP—a creeping sense of dread sets in. It’s the classic IT intuition: something’s off, and it smells like DNS.
Turns out, while the L1 is on the phone, an end user (who just happens to be the owner’s son, armed with admin credentials) decides he’ll take matters into his own hands. He references an ancient internal document and begins manually setting static IPs and DNS on every workstation he can get his hands on. As u/RenderedKnave wryly noted, “to his credit, he did RTFM, it's just that the FM was F'n wrong.”
The Ancient Documentation Menace
If you’re groaning at the mention of “ancient internal doc,” you’re not alone. Commenters zeroed in on this classic pitfall. u/OldGeekWeirdo chimed in: “Let this be a lesson - purge outdated docs.” Several others echoed the near futility of this task—once a document exists, copies will haunt you forever, popping up at the worst possible time. As u/Honest_Relation4095 put it, “You may purge them from known locations, that doesn't mean someone still has a local copy or even a printout.”
Of course, the real nightmare began when the OP finished the migration and decommissioned the old server—the very one still listed in that ancient doc as the DNS server. Suddenly, the entire office was dead in the water. Static IPs were conflicting, DNS lookups failed, printers went dark, and even a Linux box rebelled. As OP later clarified, it wasn’t just a matter of resetting 20 PCs: “IP conflicts, printing was fucked, internal lookups fucked, one workstation network stack became completely corrupted somehow, [and] they have some obscure version of Linux on a shop PC that didn’t accept typical commands.”
The DNS Blame Game: It’s (Almost) Always DNS
Within the IT world, DNS occupies a mythical status as the root cause of mysterious outages. u/sqfreak summed it up with the classic refrain:
"It's not DNS
It's never DNS
It was DNS."
Countless commenters shared similar war stories. u/faithfulheresy recalled, “Literally first 15 seconds of fault finding and I'm going ‘It's DNS,’ and everyone looked at me like I'm a madman… Two days later they figure out that it was DNS.” It’s a tale as old as time—or at least as old as TCP/IP.
But how did it get this bad? Well, as u/SemtaCert asked, “How does the end user have access to change IP and DNS settings?” The answer: nepotism’s finest—admin rights for the boss’s son. OP confirmed, “It was the owner's son who's also an employee so he had an admin password.” The consensus? Privilege management matters. As u/thevoidhearsyou shared, “Eventually got the go ahead to change everyone's privilege level who wasn't IT or management… Mr I knows better screams he can't change anything.”
Lessons from the (Comment)ariat
Beyond the comedy and collective groans, the Reddit community surfaced some key lessons—sometimes with a shot of much-needed humor:
- Purge Old Docs… If You Can: Outdated documentation is a lurking danger. But as commenters noted, truly eradicating them is nearly impossible. Someone always saves a copy, makes notes, and spreads the legacy of chaos.
- RBAC Is Your Friend: Role-Based Access Control isn’t just an acronym—it’s a survival strategy. As u/Glitch-v0 quipped, “Truly RBAC was lacking.”
- Don’t Set Static IPs in the DHCP Range: u/ponakka highlighted how this compounds the mess, creating IP conflicts that fester until everything explodes.
- Communication Is Critical: OP’s big mistake? Assuming the message to “please stop touching DNS” had actually reached the right ears. “Didn’t hear any more so assumed (big mistake) the message got through.”
- It’s Never Just the Workstations: The aftermath took hours, not minutes, to fix because DNS chaos ripples through printers, servers, and every other network-dependent device.
And for those wondering if there’s a magic bullet, u/TinyTC1992 suggested spinning up a DNS server with the old IP to temporarily resolve the mess. But as OP noted, without remote management installed, there’s no quick fix—just a lot of legwork and a painful lesson learned.
Final Thoughts: Don’t Touch That Dial (Or DNS)
So, what’s the moral of our story? Don’t touch DNS unless you really, truly know what you’re doing. Outdated documentation is more dangerous than it seems, misplaced admin rights can bring a business to its knees, and yes—it really is (almost) always DNS.
Have your own DNS disaster or network horror story? Share it in the comments below, and let’s commiserate—or at least laugh through the pain—together. And remember: the next time someone says, “It can’t be DNS,” check anyway. It probably is.
Original Reddit Post: Please don't touch DNS