How to run a simple postmortem
The world is full of imperfections and opportunity, so everyone deserves a chance to reflect and refine. This reflection and refinement can come in many packages, but the process that has helped me the most is the postmortem practice that we’ve built at work. It is deliberately lightweight and simple enough to be used whenever and wherever we would like to see better outcomes from common (or uncommon) situations. Thankfully, well-intentioned people [especially me!] will always make mistakes, miscommunicate, and be imperfect, so we have plenty of source material. This is as true in IT as any other department, so we tried to teach our colleagues about how we did postmortems inside the IT organization by inviting a wider and wider set of colleagues and by using a consistent meeting framework that helped create a rich dialog and incremental improvements.
It is now relatively common for someone to say “this is worth a postmortem” and I’ve been surprised how many of my favorite postmortems have nothing to do with an outage or an “IT incident” at all. Three examples:
- After switching new product development to small, autonomous, mission-driven teams, we held a postmortem with VPs of Product, Engineering, and Sales to capture how it had unfolded and pick one way to improve the relationships between the teams and other departments.
- We held postmortems inside the technology management team after an important employee gave notice and also after a new hire wasn’t a good fit.
- A marketing campaign to invite a colleague caused a bigger impact for our Sales team than we expected, so we held a postmortem about how we selected experiments and communicated hypothesis with our Growth, Engineering, and Sales leaders.
Each postmortem creates an artifact and one binding commitment. The artifact is a written version of what happened (from many perspectives) and why and the commitment is a single concrete, assigned, achievable systemic improvement.
This is our template, the script we paste into the minutes for every new postmortem:
Human-Readable-Name-Here (Postmortem date here)
Event timestamp:
Meeting timestamp:
Who:
The facilitator must start the with a summary of the structure of the meeting, clarify the goals, declare the meeting a safe place, encourage everyone to participate (both in helping with the notes and in talking).
What
This is a factual replay of the things that happened (with links and quotes if possible). This part is calm.
Discussion
This is where we talk about what we think about how and why and what to do. This part is more emotional, but we treat everyone with respect and do not allow blaming or shaming.
Action item ideas (non-binding)
- (A number of items)
One thing we’ll do right now (binding)
- (Only one thing)
Facilitate carefully
If you’re establishing postmortems for the first time, find a senior person inside your organization who truly believes in the merits, understands how to create safe and candid meetings, and is willing to spend the time practicing. They’ll need to facilitate each one until the basic rhythm and ground rules are internalized by a majority of the regular attendees. After that point they should cultivate a handful of other facilitators to start scaling the process.
If you’re growing an existing postmortem practice outside of one department (almost always IT), invite more and more outside-the-department people to postmortem meetings and start expanding the scope of events you deem worthy of a postmortem. In my experience other people are actually hungry for an achievable, non-depressing way to reflect on past work, so you may get some other champions fairly easily.
One caution: be prepared to volunteer experienced facilitators for a long time as the practice spreads — it’s easy to have one of these meetings veer into “blame and shame” and you should be vigilant about ensuring that the first experiences for each new department are constructive.
The best facilitators have at least a basic understanding of human bias and social psychology. If you can spot “Fundamental Attribution Error” in a conversation and gently point out this bias, you might be a good fit.
Create an artifact for many
A goal of your postmortem practice should be clear, asynchronous communication, so collect all of the postmortems in a single document for the whole company. Put the most recent at the top and keep the template as the preamble to make it easy for others to initiate a new one. [We keep ours in a Google Doc and start a new one each quarter to make the size manageable].
You can always decide to hold a private postmortem with private notes (using the same template) for sensitive situations, but the default should be public. [We put the aforementioned postmortem about an employee giving notice into a private doc, for example.]
Use a clear name for the title of the postmortem to help other people (including those in other departments) find or scan postmortems. [“Google Analytics analysis errors for product name” rather than “GA error BI-2354.”]
Use the date of the postmortem in the title to make it easy to keep them in last-first order (you’ll note the event details separately).
Capture small details
Try to note the time range or timestamp of the event being discussed precisely in Event timestamp, as some people reading in the future might be confused if it’s an event that happens again.
List the Meeting timestamp just for housekeeping (sometimes you want to analyze the delay between events and postmortems).
We write down Who: to make sure we understand the different perspectives that have been woven into the narrative that follows.
Create a space for difficult discussion
As facilitator, you have a significant responsibility: maintain a safe dialog (to ensure contributions from everyone), forbid disrespect of anyone, but get people be very frank and candid (often about their own mistakes or the mistakes of people they like). I’ve found it easiest to share and remind people of these rules at the very end of the introduction. You absolutely need to talk about this out loud (don’t rely on people reading the notes).
I start the meeting by clarifying the specific event we’ll focus on (often explicitly excluding some related aspects or events to help narrow the focus), talking about the three phases (what happened, why it happened, and what one thing we could change to create a better outcome if it happened again), and giving my sincere speech about candor and respect.
Also make sure to remind people that anyone not talking should be helping to write the notes, as the written record is one of the key reasons for the postmortem in the first place.
Collect a more complete version of what happened
The postmortem needs to capture an event from the perspective of many players. None of us will have a complete version of the event so we need to weave together a more holistic narrative from the threads we each have.
I usually say these words aloud:
This is a replay of the things that happened factually (with links and quotes if possible). This part is calm.
The “What” section is where distributed teams often shine (or those that have adopted chat tools like HipChat or Slack). You should have lots and lots of factual accounts from different voices. Paste the most interesting quotes from those chat discussions into the notes. Find relevant screenshots, emails, Google Docs, comments from project tickets, calendar events, or handwritten notes to add. You want a rich patchwork with a blend of narrative and hard quotes.
Make sure that you focus on dates and times and the context of the way the events unfolded. This has two purposes: it establishes a clear sequence and it warms up people’s memory. Our human memory is fairly buggy, so one person’s internal narrative often distorts the sequence. A recent postmortem included a person being sure they had struggled with a problem for a number of days before making a change, but a careful review of the calendar and project tickets showed that they were actually out of the office the week before. This is fine. S/he didn’t get “in trouble.” They were sincerely recounting what they remembered.
Be sure to ask the people you know have a different perspective for their version of the timeline (even if they were only involved in a small piece of it).
Consider how and why
Creating the consolidated narrative usually takes at least half the time and often the group will naturally switch into thinking about what to do next and how this came to pass. If you’re running out of time (perhaps 1/3rd left), you should wrap up the first section explicitly. In both cases, I usually say these words aloud:
This is where we talk about what we think about how and why and what to do. This part is more emotional, but we treat everyone with respect and do not allow blaming or shaming.
In most postmortems, this is the section where people are most eager to contribute. They would love to try to solve the puzzle of the root cause or speculate on why the events unfolded the way they did.
Encourage many possible improvements
The transition from telling stories about why something may have happened into what to do next may happen naturally, so if you’re the one taking notes jump back and forth between this section and the before, adding rough items to a bulleted list to be refined. When you’re ready to switch to building a list of potential candidates for the final, systemic improvement, say each of the initial ideas (and give credit to the people who thought of them). Encourage people to build on top of existing ideas in addition to posing alternatives.
You should stress that these are ideas, not binding commitments (to make sure you get suggestions from a wide group). “Action Items” may be too overloaded a word for some organizations, so choose a less rigorous alternative if necessary. You will pick one by the end of the meeting and you should assign it to a single person for accountability, but too often postmortems are a time when teams overpromise and overcommit. Don’t do that. If an individual is so motivated by an idea from the meeting that they go and do it on their own volition — great, just don’t force them.
Pick one improvement, but you’ve got to actually do it
By the end of the meeting, the facilitator or person who called the postmortem needs to pick the single binding outcome. There is often value in using the time pressure of the end of the meeting as a constraint to improve good ideas and pick just one. This part is more art than science, as you need to balance achievability with effectiveness.
The best ideas usually fall into two groups:
- Changes that will make the error/incident/whatever less likely in the future
- Changes that will make the error/incident/whatever less painful in the future
When in doubt, steer toward the second group, as humans have a tendency to think they’ll be able to be perfect in the future.
Another filter for selecting the best improvements is to focus on a change that will help a broad range of incidents a little rather than a significant improvement for a narrow subset.
A warning: if you’re building a postmortem practice for the long term you’ve got to make absolutely sure the “one improvement” from earlier postmortems actually gets done, otherwise the process will lose people’s support.
FAQ
- Aren’t these just retrospectives? Yes, that’s a different way of trying to get to the same goals. I’ve found it more helpful to use a consistent pattern with people, but it depends on the context. We don’t use a “postmortem” at the end of each sprint, for example.
- How long should the meeting be? Start with 60 minutes. With practice, you can often make them work in 45. 30 seems too short in practice, but we’ve done a few. 60, or perhaps 75, feels like a good constraint to me, as the discussion can wallow if you have too much time. I’d much rather have two 60 minute postmortems about two aspects of a significant event than try to distill everything into one 90 minute meeting.
- Should we wait until all the __________ are known before scheduling the meeting? Almost never, in my experience. We try to schedule postmortems as soon as anyone has an inkling that one might be worth it. They get much harder to run as an event gets farther in the past and memories fade. There seems to be very little downside to scheduling one promptly, as you could always have a second when some other parts unfold or conclude.
- What should I do if we start talking about why or ideas on how to improve before the timeline is complete? In these cases, I start faithfully taking notes in the second [“Discussion”] section and bring attention back to the timeline [“What”] after the person has finished speaking. When in doubt, remind folks that writing a reasonably clear sequence from multiple perspectives is a prerequisite to uncovering the most interesting opportunities for systemic improvement.
At Safari, we believe that great work is done by people who are paying attention to new thinking, who are versed in the classics of their discipline, and who share what they learn with those around them. I would be remiss if I didn’t clarify that there are many other perspectives on postmortems and that I’m far less experienced in this are than many others.
John Allspaw is an important leader in the “devops” movement and has written and talked extensively about ideas like blameless postmortems. He did an extensive tutorial on facilitating postmortems at Velocity NYC 2014, which includes a lot of the background I’ve glossed over.
As I mentioned above, taking time to study human bias and social psychology can be very rewarding when doing this sort of work. Daniel Kahneman (Thinking, Fast and Slow) is very rewarding if you’re willing to commit the time. Dan Ariely’s writing is more accessible (and comes in much smaller packages).
Dan Milstein has given various talks and written on postmortems and “5 Whys”. I particularly like the way Dan is able to exploit financial framing to get us to think differently.
While some of the thinking about how to think about human bias and complex systems has changed in the last decade, postmortems and retrospectives are a very established discipline, especially for big (software) projects. Project Retrospectives provides a more extensive description of how to think about and run these meetings (including exercises).
If you’d like a longer discussions about postmortems in practice or have questions, find me on Twitter or email.
