The Case of the 7:00 PM Network Outage: How Stadium Lights Took Down an Oilfield’s Internet
Ever had your internet disappear at the exact same time every day? Imagine if it wasn’t just your Netflix binge being interrupted, but a whole section of a company’s network vanishing like clockwork. That’s exactly what happened at an oilfield service company’s remote training site—and what followed was a tech support mystery worthy of Sherlock Holmes (if Holmes had a CCNA, that is).
It all started, innocently enough, with an oddly punctual outage. Every night at 7:00 PM, a chunk of the building’s network would drop off the face of the earth. No emails. No file shares. Just the digital sound of silence. And no amount of remote troubleshooting could shed light on the issue—until someone noticed what else was happening at that magic hour…
Let There Be (Too Much) Light
Let’s set the scene: It’s the early 2000s, and our hero is a member of an IT support team responsible for a sprawling oilfield operation, including five remote sites. One remote facility—a training center—is acting up. On the dot, at 7:00 PM every evening, part of the building’s network evaporates. The team’s remote tools can’t find anything amiss, except that the distribution switch down that wing is suddenly unreachable. The core switch is fine. The other distribution switches are fine. It’s just this one, and always at 7.
So, the boss grabs his troubleshooting kit and heads out. He waits. And right as the clock strikes 7:00, POOF! Connectivity to that wing vanishes. All hardware is powered and appears operational, but the core and distribution switches refuse to talk.
This is where the story shifts from mere annoyance to full-on tech support detective work.
The Clue in the Darkness
The next day, the boss returns to the scene, determined to observe everything that happens as the outage occurs. He watches as dusk falls. And then he spots it: The security guard, beginning his nightly patrol, flips the switch on a row of massive, stadium-style floodlights—newly installed and bright enough to melt retinas.
And, right then, the network goes dark.
The Real Culprit: Ethernet vs. Electricity
Now, any seasoned network tech’s spidey-sense would be tingling. The boss investigates how these new lights were wired in. Turns out, the contractor responsible for installing the lights decided to save time (and a buck or two) by running the high-voltage electrical cable right through the same duct as the network’s Ethernet cables.
If you’re not familiar with the finer points of networking, here’s the scoop: Ethernet cables are surprisingly sensitive to electromagnetic interference (EMI), especially from beefy power lines carrying enough current to light up an industrial site. The moment those lights powered on, they bathed the Ethernet cables in a soup of EMI, effectively jamming the network signals and severing the digital connection between switches.
The Fix? Simple, Once You Know the Cause
Once the boss had the cables separated—network in one duct, high-power lights in another—the outage vanished. The network ran like a dream, no matter how many stadium lights blazed.
This story isn’t just a fun trip down memory lane. It’s a classic lesson in “layer 1” troubleshooting: The physical layer. As much as we love to blame software, firewalls, and configuration files, sometimes the answer is hiding right behind a wall—or, in this case, inside a duct.
Lessons from the Field: Why Tech Mysteries Matter
There’s a reason so many tech support war stories end with someone physically visiting the site. Remote troubleshooting can only go so far. Sometimes, you need to see what else changes when the network goes down—and that might mean noticing someone flipping a light switch or plugging in a microwave.
And let’s not forget: shortcuts in cabling or infrastructure might save money in the short run, but they often lead to bigger headaches down the road. (Fiber optic cables, as this oilfield eventually upgraded to, are immune to this kind of electromagnetic drama—take note, future contractors!)
Have You Faced a Timed Tech Mystery?
Network outages with a precise schedule are rare, but when they happen, they’re as satisfying to solve as any locked-room mystery. Have you ever cracked a tech problem with an unexpected culprit? Ever found a printer that jammed only when the coffee machine was on, or a WiFi black hole under a disco ball? Share your own tales of tech troubleshooting in the comments—let’s keep the “aha!” moments coming!
And remember: If your network vanishes at 7:00 PM, check the lights… and maybe the ductwork, too.
Original Reddit Post: 7:00 PM Network Outage