Find your sigh
Dealing with production support
“Oh dear.”
It was one of M’s catchphrases and when you heard it, it was showtime. It always had the same singsong, resigned lilt, as though he had just looked into the sock drawer and found them incorrectly sorted. But it usually meant that something was very wrong, probably in production.
Production support is a stressful responsibility, and is one that many technology workers assume at some point. After twenty years, I’ve learned that being good at production support is less about knowing everything and more about being able to cope with the stress. I’d like to share some of the techniques I’ve picked up over my career.
“Oh dear.”
Something was broken and it was affecting our customers. I knew at this point that there would be some kind of quick fix. It might be a tweak in production. It might be a software patch that had to be deployed or a rollback to a previous version. Perhaps we had forgotten a database update. Whatever it was, M would quickly characterize it, let others know what was going on, get any help he needed, and get the fix in. I’d been doing production support for a decade or so by then, and I’d never seen someone manage it so calmly.
The “Oh dear” was also catchy, and before long we all started to say it when something went wrong: a leitmotif occasionally emerging over the drone of mechanical keyboards. When I picked up the phrase, I noticed something interesting. The simple act of saying it relaxed me. I’d always gotten anxious during production support incidents. My heart rate accelerated and I developed a plunging dread in the pit of my stomach. That initial reaction remained, but as soon as I said “Oh dear,” I would settle. I could still feel those emotions, but I could control them and slow myself down to focus on the task at hand.
Since then, my “Oh dear” has evolved into a long, melodramatic sigh, but it serves the same purpose. When something goes wrong, I sigh audibly — many would say comically. After the sigh, I am centered and focused on what needs to be done.
That sigh, based on M’s “Oh dear,” is probably the single most important production support technique I’ve learned. It’s always stressful to be pulled into a production incident. That will never change; I still get anxious when it happens. But the sigh calms me and helps me to be more effective in the moment. It also helps to lighten the mood right at the start. The situation is still urgent, but I’ve blunted the sharp edge.
If you find yourself responsible for production support, find your sigh. Do something physical and audible to pull yourself out of the moment and settle your nerves. It could be a sigh, or a catchphrase, or a whistle, or a show tune. It doesn’t matter what it is, as long as it’s audible and perhaps a bit silly. The anxiety may never go away entirely, but you’ll find yourself better able to set it aside and deal with the situation calmly, which is the most important consideration when dealing with a production incident.
