What does it mean to debug the hard problems?

In my opinion, one of the hallmarks of being a “senior” engineer is the ability to debug the hard problems. The problems that span systems, some of which are likely external or black boxes, and most assuredly lacking documentation. Ones that span the stack, requiring knowledge of the client, server, and the network between. The kind of knowledge that can double your throughput with a config switch. Here’s a great story from the Navy that demonstrates this skill in a non-software environment:

It’s World War Two in the Pacific. The US is fighting a desperate battle against Japan. US submarines coming home complain that their torpedos aren’t working. The torpedo manufacturers say they are fine. You’re in charge. What do you do?

1. Tell your people to get er done?
2. Tell your people to bring you “solutions, not problems.”?
3. Quote “Message to Garcia?”

If you’re Charles ‘Swede’ Momsen, you do none of these. You take a bunch of torpedos, shoot them into a Hawaiian island, find one that doesn’t work, dive in yourself to save it, and figure out that the firing pin wasn’t igniting properly.