Ask 5x why? I would rather die
I wouldn’t rather die. But it rhymed, so there you have it.
When stuff goes wrong, we have this strange notion that we can prevent stuff from going wrong in the future. Which is funny, since something just went wrong, and we failed to prevent it.
There are six million ways to die
At any given moment, there are a million things that can go wrong, in a million different ways. The reason why it goes wrong, is so complex it’s effectively random. What no one predicted in advance, appears obvious in hindsight. Except it isn’t. You think you figured out why, except you didn’t. What you did was fabricate an explanation that sounds reasonable. In reality, you have no way of knowing whether it’s actually true, even after the fact.
Our instinctive reaction is to analyze stuff that goes wrong, so that «something like this can never happen again». I wouldn’t worry too much about that, because it probably won’t. Something else will, only you don’t know what.
Press releases stating that “we have taken steps to prevent similar incidents from occurring in the future” are at best wishful thinking, and most of the time outright lies. I regret to inform you that no one has any idea how to prevent shit from going wrong in the future.
On the bright side, you successfully prevent hundreds of things from going wrong every single day. Unfortunately no one ever notice, so you don’t learn much from that either.
So, what to do?
Don't try so hard to figure out what can go wrong, build the capability to fix it immediately when it does.
Don't stuff so much work in one release that something is bound to go wrong. Release as little as possible, as often as possible.
Automate tests, and build feature toggles so that you can test in production. This allows you to fail safe in a test environment or on a limited subset of your users.
Run retrospectives. Make sure people feel safe to discuss without fear and with the intention to improve.
In conclusion: Shit will fly
Figure out how to intelligently handle brown fans. Because there will be brown fans.
