How to do a Retrospective using Flow Metrics

After having read Dan Vacanti’s book on Actionable Agile Metrics for Predictability I decided to put my learnings into practice and teamed up with a Scrum Master of a team that recently started working with ScrumBan. By simply looking at their Kanban board it was apparent that there were plenty of slow running tickets that were stuck in the Implementation phase. To improve the team’s predictability (“When will it be done?”) we both decided to bring this issue to the team.

First, we used jira-agile-metrics, a Python tool, to extract flow charts out of the data provided by the team’s Jira project. Then we prepared the workshop using the generated charts.

Setting the Stage

To get started, we introduced the Cumulative Flow Diagram in the Retrospective and asked the team “What stands out on this chart?”.

Cumulative Flow Diagram

The team quickly identified that the Work in Progress (WIP) is steadily increasing over time, meaning the team is starting more work than they are able finish. In short: this system is unstable as the input is increasing over time and the output stays the same.

Cumulative Flow Diagram

The next chart we showed the team, was a Cycle Time Scatter Plot that visualizes the Cycle Time for each ticket in the Done state. According to the chart, the current 85% percentile of the Cycle Time is 18 days. This means that out of 100 tickets 85 were finished within 18 days. Furthermore, the chart indicates a pattern where tickets are usually closed every two weeks.

Cycle Time Scatter Plot

After that, we introduced the Ageing WIP chart that showed the snapshot of all the current tickets that are not done yet. “What do you notice? And what’s the effect on the Cycle Time?” we asked the team.

Ageing WIP

The team pointed out all the blocked tickets in the Implementation phase and concluded that as soon as all the tickets, which are currently aged 18 days and more, will increase the Cycle Time, as soon as they are moved to Done. To set a specific focus for the Retrospective, we narrowed down the selection and targeted all the slow running tickets that were open for 25 days or more.

Tickets open for more than 25 days

Gather Data

All team members paired up and got each a couple of tickets to analyze. The initial task to answer the question “What’s preventing us from closing this ticket?” without providing solutions for each item. Each group briefly described ​to the team the reasons that hinder the tickets from being closed.

Generate Insights

After that, we discussed the patterns that became visible. The ones that we found were:

  • Priority of the ticket was lowered after having started with the work
  • Work was started too early on the ticket
  • Waiting for external dependencies (individuals, teams or supplier)
  • Requirements not clear
  • Requirements changed after having started working on the ticket

Decide What to Do

We brainstormed different ways on how to eliminate the unnecessary waiting time in our tickets. In the end, we decided on three tangible action points:

  • Introduce an Analysis state in our Kanban workflow to prevent starting too early working on tickets. In this phase we want to clarify scope, external dependencies, prerequisites and size.
  • Introduce a Parking lot to place tickets, where we are waiting for external dependencies without much leverage
  • More focus on the data quality of our Kanban board by updating the tickets more often

Closing

To conclude the Retrospective, all three improvement tasks were created and added to the team’s backlog.

One Month Later…

… from a WIP perspective not much has changed since: the team’s Work in Progress is not limited and is steadily increasing.

Work in Progress

After checking back in with the Scrum Master, he shared the following observations with me that are partly visible on the CFD as well:

  • The Analysis phase in the Kanban workflow has been implemented, WIP is limited for that phase, and is in use by the team.
  • The Parking Lot has been implemented as well and the number of tickets is stable
  • There is increased awareness in the team to keep Jira ticket quality high and a discussion about the scope (in scope/out of scope) and the Acceptance Criteria of the tickets, driven by the team members.
Cumulative Flow Diagram

My Learnings

Preparing and facilitating this Retrospective together with the Scrum Master was very insightful:

  • I have learnt that framing the topic of the Retrospective to the team is crucial “Why are we doing this Retrospective about flow?”.
  • Having the team understand that flow charts are not about measuring and comparing their performance but all about having a tool to ask the right question and setting a focus for inspection and adaption.
  • The flow charts are only as good as the data beneath in Jira.
  • It is challenging and open for interpretation to determine if a measure was successful. Next time, I will formulate a Test Card for each improvement item (hypothesis, test, metric, success criteria).
  • Last, improving the predictability is a continuous task. A one shot inspection and adaption has no sustainable effect.