Engineering, and asking the right questions
Most software engineers are well-versed in the following stages of problem solving (•)
- Problem Recognition: aka “Houston, we have a problem”.
Observability is all well and good, but you need to know that what you’re observing is not a Good Thing.
On the one hand, the mundane stuff gets easier over time, as you add in monitoring (and self-repair, right?) for crashes, failures, overloads, and the like.
On the other hand, “black swan” syndrome kicks in — as you squeeze out all the common stuff, you end up dealing with weird/edge-case problems that you don’t even notice till all hell breaks loose at 3am X-mas eve. - Fault Identification: This is where you figure out what exactly happened, and is, hopefully, something straightforward like “the node crashed”, “we’re being DDoSed”, or some such. It is also usually the easiest part of the process (once you get past all the alarms, flashing red-lights, and panicked calls, that is!)
- Asking the right questions: And this, boys and girls, is the really really hard part. You know you have a problem (“The service is down! Down I tell you!”), and you know what’s causing it (“The job queue is backed up! WhyTF is it backed up?”), but you have no clue what’s actually causing it.
The good news is that you have entirety of Google, StackOverflow, and Twitter at your fingertips. The bad news is that you don’t know what questions to ask.
A huge part of your career growth as an engineer — software or otherwise — is about -3- above, i.e., learning to ask the right questions. And about the only way to get there is experience, and breadth of experience at that. The more industries you’ve worked in, the more areas in the tech-stack(s) you’ve fiddled with, the time you’ve spent on-call, the documentation you’ve written, the firmware you’ve had to debug — they all add up. And yes, a lot of this eventually ends up being “intuitive”, background wisdom that you don’t even know you know till it becomes necessary.
Incidentally, your failures are equally important here. The more mistakes you’ve made and the more you’ve f**ked up, the more chances you’ve had to learn what doesn’t work. The ability to eliminate possibilities is key, it helps narrow down the problem space!
So yeah, treasure those new experiences, and don’t be afraid to screw up. The good news is that there is always an opportunity to do both in the world of software engineering — heck, there will probably be a new JS framework tomorrow!
(•) Yeah, this is about solving the problem, not about implementing the solution 😆
