When we start learning about AGILE and SCRUM, we inevitably learn about velocity, this metric that allows us to create a baseline target number of Story Points that the team is able to accomplish in a sprint. It supposedly allows us to make long-term estimations and plan future sprints.
It was only a few years later when leading a talented team of software engineers (shout out Farouk & Sam 🐍) that I realized how flawed this metric was and found what I should have been using all along.
Why Velocity doesn't work
Let’s start by seeing how velocity is computed:
- Actual Velocity: Total Story Points Completed by the Team / Number of Sprints[1]
- Expected Velocity: Total Estimated Story Points / Number of Sprints (Estimated story points being those added to the Sprint)[1]
In practice, the Actual Velocity is the one most team use and the one we’ll be basing ourselves on.
But how is Actual Velocity flawed?
Variability
When computing Actual Velocity, the variability of what is actually completed is never taken into account. See the following examples with two teams using the same scale for story point estimation:
Even though both teams have the same velocity, can we confidently say that both teams will be able to complete 20 story points per sprint?
For Team 2, maybe, the variability between the number of story points per sprint is very low (+/- 2 Points), but Team 1’s completion variability is through the roof (+/- 18 points) so we cannot be confident that they can consistently complete 20 story points per sprint.
Thus can we confidently estimate future stories and tasks, or give an estimated release date for an epic?
We can’t.
Inflexible to change
Teams and requests often have change: team member on vacation, team member onboarding, urgent bug fix, company training, etc…
The Actual Velocity is calculated based on the theoretical average team functioning but will the team be able to complete all the stories if one of the members is away on PTO? What is the impact on our velocity? Will the team be able to accurately estimate their story point capacity?
Or conversely, if we have a new joiner on the team, should we increase our velocity since we have a new team member? Should we decrease it for this sprint as they are going to take time to train? Will we need to re-estimate our backlog? We have no clue.
Estimation
At the end of the day, Velocity does not help teams estimate better as it relies on them to make an accurate Story Point size estimation for stories and tasks. Story points are made up, they should help the team estimate the stories and tasks. But due to that made-up nature difficulties arise when the team tries to estimate the size of a story.
So how are we supposed to achieve a consistent velocity if we are unable to accurately estimate our stories and tasks? How are we supposed to understand how this will affect future estimations? How can we know if our method of story point estimation is too granular or not granular enough? We can’t.
Burnout
Finally, setting a team’s sprint goal based on their velocity can and will inevitably lead to burnout. We are all familiar with the work crunches, we need to hit a deadline so we overload the sprint backlog, adding all the tasks that need to be done and making sure that we complete them all even if it means working 15h days or working on Saturday and Sunday to reach that goal.
Hopefully, these events don’t happen often, but they set a precedent. They increase the Actual Velocity as the team has just completed a sprint with more story points than the average.
So if we previously had a velocity of 20 point per sprint, we might now be at 21 points per sprint. If we perform lower than a certain amount compared to our actual velocity, the team might be impacted so we will do our best to achieve that 21 points per sprint goal until the next crunch comes along and our actual velocity is now 22 points per sprint, etc…
Velocity is not used by teams to set a goal for themselves, but by managers to set a performance precedent and expect future value.
If you’ve read this far you’re probably saying to yourself: “Ok cool story bro, but what do we use instead?”.
Commitment Variance
The Commitment Variance metric allows teams to self-organize and self-direct.
I calculate it as follows:
Where
- PointsCompleted indicates the number of story points completed within the last sprint
- PointsCommitted indicates the number of story points the team committed to (i.e. added to the sprint backlog)
Goal
The goal of the team, using this metric, is to accurately estimate and commit to a set of Stories and Tasks (and complete the all, and have the burn-down chart reach 0 at the end of the sprint).
Interpretation
If the Commitment Variance is
- Greater than 0: then the team has over delivered. This is great and they can use this information to commit to the same or more story points next sprint depending on which factors may have caused them to overdeliver (e.g. over-estimated stories, unexpected addition to the team, etc….)
- Less than 0: then the team has overcommitted. This may be difficult, but let's use this baseline to identify what caused the team to over-commit and realign with the commitment next sprint
- Equal to 0: the team has accurately estimated their stories, tasks, and commitment and seen it through. They should continue forwards in this way!
Through time, a team aims to lower the absolute value of their commitment variance to as close to 0 as possible.
Benefits
By using this metric, the team:
- Decide their commitment and goal for the sprint. This allows them to be flexible based on any team changes (PTO, New joiner, transfer, risk of ad-hoc requests), etc…
- Focuses on accurately estimating the tasks. The team will be able to properly leverage and define what story points truly represent for them to estimate tasks properly.
- Focuses on accurately estimating their capacity. By allowing them to choose their commitment, they can identify and communicate what they know their estimated capacity to be.
- Reduces risk of burnout. Work crunches will happen, but burnouts shouldn’t. By allowing teams to make commitments that are reasonable to them, you are allowing them to mitigate the risk of burnout by themselves.
And for the leaders, this allows them to have the:
- Increase confidence in the teams’ estimations. By knowing that the team can accurately estimate the stories, tasks, and their capacity, you can work closely with them to estimate the complexity of a new feature and/or product.
- Report more accurate delivery deadlines. By having confidence in the team’s ability to estimate the complexity of requests, you can give more accurate delivery dates keeping stakeholders happy.
- Give ownership to the team. By allowing the team to self-organize and target the Commitment variance goal of 0, you are giving them ownership of the work they have committed to which can help motivate them.
Risks
As anything, using this metric incurs some risk:
- Self-pressure. The team may feel internal/personal pressure to overcommit and over-deliver as they may feel they are judged on it. It is a leader’s responsibility to create a psychologically safe environment where teams and team members don’t feel that pressure. As a team leader, you must also use this to identify if the team is pushing themselves to overcommit which may lead them to burnout.
- Under commit. Some teams may find the freedom of choosing their capacity and commitments allows them to under-commit and deliver just what has been committed. It is a leader’s job to identify if this pattern does start to happen and identify ways to mitigate this risk.
Next steps
Only once you have started using commitment variance should you try and use velocity to get a baseline and use it for your estimations. Supplements that velocity by adding a buffer to your estimation based on the commitment variance you’ve calculated.
Example Scenario
Let's go back to our previous two teams, now assuming they use Commitment Variance to assist them in estimating their stories, tasks, and sprint capacity.
The average of the teams’ completed story points gives us our Actual Velocity and looking at the teams’ committed story points, we see that both teams averaged close to 22 (+/- 0.25) story points per sprint. But their actual work completed is very different.
Let's give more information about Team 1. Using Commitment Variance, they identified that the current way they were using story points was not allowing them to estimate their work and capacity properly.
In sprint 4 they redefined the way they define and use story points to estimate, adding much more granularity, leading them to commit 28 points even though they only completed 12 in the previous sprint.
Graphing their commitment variance, we can see that this showed a constant improvement irrespective of how the Story points are defined.
Will you start using Commitment Variance with your team?
What other benefits do you see using this metric over Velocity?
What metric do you prefer to use over Velocity?
