3 key ingredients to building a strong incident management process

1. Define what an incident is

This is, in the words of Julie Andrews, a very good place to start.

  • Don’t assign blame (or other negative consequences) to declaring an incident. This includes not using ‘number of incidents’ as a metric for team health or performance.
  • Your lowest severity should represent something that happens regularly and has a low impact, so people feel comfortable declaring incidents (and don’t feel like a boy crying wolf)
  • Don’t limit the definition to engineering: getting your whole org to think about anything unexpected in the same way means you can collaborate better (see section 2)

2. Keep everyone updated

Transparency by default is a really important value to bake into your incident process.

3. Remember: it’s not over until it’s over

Your incident process shouldn’t end once the problem is resolved. To get value from your incidents, you want to be using them to learn and improve your day-to-day operations. There are often follow-up actions that need to be put on someone’s backlog or wider problems that should be considered and prioritised.

Summary

When you’re first building an incident response process, focus on a few key things:

  1. Make it easy to provide updates, both to internal and external stakeholders
  2. Reflect on incidents to gather learnings, but make sure the process is adding value

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
incident.io

incident.io

Incident response for fast-growing tech companies.