Planning Ahead — The Tale of Network Switches and a Thunderstorm

Often times your network is so far removed from your thoughts because it is always there, always doing its job. When issues occur with your network though many things can go very wrong. You quickly realize how much you and your organization relies upon a quick and responsive network. What follows is an anecdote regarding what could have been a disaster but ended up being only a slight nuisance.

One weekend a few months back, some sections of the east coast were experiencing thunderstorms and it was a quick reminder of how some forethought into the architecture and design of your network can pay off in big ways. Our team was alerted to an outage with one of our clients. Some quick troubleshooting found that some switches in a switch stack had failed.

They were, as I like to call them, dead dead. No lights, no noises, just a large expensive hunk of metal.

Ultimately the incident required some forensics to discover how or why this equipment was harmed while being behind so many power safety devices. At the time, though, the most important thing was to right the capsizing ship. This particular stack of switches served as the core of the network and was made up of 5 Cisco 3850s. It was at this very moment that the client recognized the thought put into our design early on two years prior.

You see, we had placed a spare switch both within the rack and the switch stack. This meant the unused switch was always at the ready. Physically it was in the middle, this meant that cables from any of the other switches could reach the switch and it also meant that making configuration modifications were just as quick. We talked the client through moving cables, port 1 from the dead switch moved to port 1 on the spare, port 2 to port 2, and so on. We could easily copy the configuration dead switch port configurations and within 15 minutes we had everything back online.

Following the emergency, the dead switches were replaced via RMAs within another two days. Our team was excited to work with the client in a moment of stress with calm instructions, knowing that planning had paid off. What could have been a stressful endeavor filled with upsets, was instead a routine process executed by a confident team.