My Love-Hate Relationship with Agile Metrics
A valuable insight into more fitting metrics for your team
Metrics are a hot topic in many (Agile) organizations resulting in strong opinions and heated discussions. In my experience, when management talks about metrics they often think about holding a team accountable and having to justify the work done. It is often seen as a reporting tool.
In situations where metrics are perceived as ‘there for the management’ more than for the team itself, there are missed opportunities. Far better reasons to use metrics are to achieve and sustain continuous improvement, to make predictions and predictability easier, and to motivate and adhere to the responsibility of the team.
So, how do you use metrics in a positive way that speaks to the team and gives insights into value and improvement?
I once was working with a few teams where I took some time every other Friday to work on dashboards. I started with metrics about the number of committed user stories and achieved velocity, but quickly built out metrics like lead time, the number of bugs, code quality, and team happiness. It became a hobby to see what I could measure, not thinking about if I should measure it. The more I tweaked, the less involved the team became with the dashboard; this frustrated me.
After discussing this issue with my teams I learned (the hard way) that metrics should support the team and I should start with questions like; Why do we want to measure this? What are we going to do with the results?
The next day I threw away 80% of the metrics and started over. In the years after I kept my dashboards small and very much ‘in line with the comfort level of the team’.
I would mainly focus on forecasted-story-points and achieved-story-points, resulting in a velocity (5 sprint average) and a predictability (percentage of forecasted story points realized, a score of 80% and up realized would result in a successful (green) sprint).
I help my teams understand that predictability is an important attribute since it can convey trust to the PO and stakeholders. Velocity should be used more for long-term release planning than for committing a number of story points in the next sprint. This blog remains one of my favorites to stress this point.
Performance Metrics vs Performance Predictors
It wasn’t until I followed a DevOps Fundamentals training a while back, that I was properly introduced to the theory behind Performance Metrics vs Performance Predictors (Lagging Indicators vs Leading Indicators). It helped me understand why and how I would want to measure something.
In this training the difference is explained as follows: Performance Metrics are typically output-oriented, easy to measure, but hard to improve or influence. These metrics are known as lagging indicators. Example: measuring weight on a scale.
Performance Predictors are typically input-oriented, hard to measure, and easy to influence. These metrics are known as leading indicators. Example: calory intake and calories burned.
Source: Training content to the DASA DevOps Fundamentals training by ISES Computrain
The dashboard I previously used was filled with lagging indicators. It may have been more of a reporting sheet, than a tool to coach and improve my teams. The dashboard pointed out bad practices, rather than help prevent or improve these issues.
Finding the right performance predictors is difficult. You should decide with your team what it is you want to measure and how you can best do so. Sometimes it is just a small difference that can give an easier approach. For instance, perhaps you and your team would replace the KPI ‘number of releases this year’ with something like ‘average days between releases’ or even ‘effort (hours) it took to release’ in combination with defining the biggest CI/CD obstacles. The effort it takes to release could be a predicting value to getting value shipped to the user.
Backlog readiness
A new metric I introduced to my dashboard recently is backlog readiness. Which is a metric indicating how many sprints ahead the team has ‘ready’ user stories for (note that a higher number is not better). This metric can predict ease of planning events, more predictable and focused sprints, and ease of releases. Also, a backlog that isn’t ready enough is a cesspool for frustration in the team.
The metric is simple: Backlog readiness = sum of estimated PBI’s / average velocity
When I calculate the sum of all estimated PBI’s I maintain the rule that no estimation should be older than 3 months. The value of an estimation is not in determining a number of story points but lays in the discussion where a team explores possible approaches, outcomes, and hurdles before they pick it up. As these variables probably changed in 3 months, there is value in re-estimating together.
The outcome of this metric is a number of sprints. Together with the team, I decide what the ideal score is. Usually, the aim is to have it somewhere between 2 and 3 sprints. This means you have plenty of stories to choose from and have the luxury to perhaps group stories together to make a sensible sprint and or release. It also means you don’t waste time on stories never getting picked up or stories that need to be re-discussed because the situation has changed.
To conclude I’d say my love-hate relationship is based on the joy I still get from creating Excel graphs and the powerful tool a metric can be while realizing you should be careful what and why you measure something. Metrics can also create an atmosphere of control and accountability that limits the freedom a team experiences. Ask yourself: Is the metric saying what you want it to say?
So perhaps it is like Spiderman’s Uncle Ben said: With great power comes great responsibility.
What are your experiences and best practices with Agile metrics and when and how to use them? And what metric do you have good experience with in motivating a team?
