Why Agile Velocity is the Most Dangerous Metric for Software Dev Teams

Agile Velocity is arguably the most popular software development metric in the world. It’s a very powerful metric when used for individual team sprint capacity planning. And there are two things we know about power… it comes with great responsibility and it corrupts.

When agile velocity is used for anything other than individual team sprint capacity planning, very bad things can occur. In fact, when misused, Agile Velocity is the most dangerous metric for software development organizations. Unfortunately, every day Velocity is abused by executives, engineering leaders, product leaders and even developers.

Already familiar with the dangers of using agile velocity? Click here to see 12 alternative metrics for team-focused software development leaders

What is Agile Velocity?

Agile Velocity measures the amount of work a single team completes during a software development iteration or sprint. It represents the amount of story points completed over time and can be visualized as the slope in a classic burndown chart.

What are Story Points

Story Points are a unit of measure for expressing an estimate of the overall effort that will be required to fully implement a product backlog item or any other piece of work. Story Points are a subjective measure decided and agreed upon by an individual software development team or organization.

My personal history with Story Points & Agile Velocity

When I was promoted into my first team lead role, like most new managers (in any function), I was in over my head. But I was determined to do a great job. I looked for every opportunity to help my team improve. Data. Metrics. Process. Culture. I was hungry for all of it.

I remember thinking that, compared to other departments like sales, marketing and technical support, there didn’t seem to be a lot of established metrics engineering leaders used to measure team-based performance and improvement. Which is ironic because we measure so many things — page load performance, latency, code performance, unit test coverage, etc.

Then I found velocity. I was pretty excited. Of course I had heard of it but I never did a deep dive. We used Jira and story points but I was never trained on velocity and it was not part of our formal process to measure story points across multiple sprints.

We started using it for capacity planning. It worked! You get what you measure, right? We started measuring velocity and the focus on it allowed us to get progressively more accurate in our sprint planning.

I was promoted to lead more teams. At some point in a management meeting I asked all of our managers “Should we start reviewing velocity every week in this meeting?”. And I was actually ready to go further… scrum of scrum, all-hands meetings, etc.

I’ll never forget what happened… Tim Wall, a vet on our team who I really respected, gave me the following advice: “Never use velocity to measure performance and never share velocity outside of individual teams.”

Thankfully, I listened to Tim. Because since then I’ve heard nothing but horror stories from other engineering leaders who tried to use agile velocity for more than what it was originally intended for.

Dodging the Dangers

There are three main types of abuses I’ve seen of Agile Velocity. Consider this a hero’s guide to help the software development community stay safe, avoid traps and, most importantly, to use Agile Velocity for good not evil.

1) Agile Velocity abused by Dev Leaders as a team comparison metric

I narrowly escaped making this mistake. Some engineering leaders I know weren’t so lucky.

Here’s the scenario… Agile scrum is being used by all of your teams. Velocity is being tracked on an iteration by iteration basis. You have a gut intuition that some of your teams are “performing better” and some of your teams are “performing worse”. You get an itch to see which teams are completing the most story points per iteration and which teams are completing the least story points per iteration. You’re just curious. Then you start thinking about it in terms of who has the “best” velocity? Maybe you even show and compare velocity in your management meeting or scrum of scrums.

What can go wrong

Danger, Will Robinson! We have to be very, very careful here.

• Story point weights are subjective to each individual team
• Therefore story point weights are easy to manipulate
• Therefore developers have an incentive to inflate their points
• Therefore individual teams and developers can easily game the system
• If some teams are inflating their story points, other developers will figure this out
• From here you are only a stone’s throw away from broken trust, frustration and culture damage

Bad intentions aren’t even required for this to happen. It’s natural human behavior.

So, like Tim told me, Agile Velocity is sacred and should be used within individual teams only. Period.

Tips to navigate this situation:

There’s nothing wrong with looking at comparative performance data. That’s your job. Without benchmarks, you won’t be able to invest your time helping your people and teams that need it most. Since you shouldn’t use velocity, what metrics should you use? It depends on the business outcome you’re trying to achieve:

• If speed to value is your main goal, consider Cycle Time.
• If predictability is your main goal, look at Iteration Churn.
• If quality is your priority, Change Failure Rate and Mean Time to Restore are good.

A related question to ask yourself is what kind of culture do I want to create? One VP of Engineering I interviewed recently talked about how measuring the wrong things can backfire and hurt culture. Chris Downard from GigSmart says

“If you stack rank individual devs you create anxiety. Chris Downard, VP of Engineering, Gigsmart

One of the things I don’t like about getting individual statistics, like the number of commits you push as an individual developer, is that you’re incentivizing the number of commits they push. Not necessarily the quality. Or not necessarily tied to the business objective for delivery.

I don’t care how many commits they make. I care about the work the team gets done. For example, if a developer starts pairing [pair programming] a bunch that should not affect how we view them as long as they keep delivering value.”

He’s not the only software development leader I’ve met who feels this way. Kara Minotti Becker, a respected agile consultant, told me recently

“Members of software development teams, including engineers, are often scared of metrics… who they’re going to be exposed to.”

I’ve seen this too. If you want to create an environment where it’s all about the team, consider throwing out metrics like individual code changes and commits and just measure team-based metrics.

2) Agile Velocity abused by Executives as a performance metric

It’s no secret… executives love data. Especially performance related metrics.

One of the biggest issues that we face as software development leaders is the lack of standardized metrics describing the health and performance of our teams. Unlike all other major departments (sales, marketing, finance), we as a community lack the basics when it comes to data driven team performance.

Sales has revenue and pipeline and sales cycle length

Marketing has MQLs and SQLs and cost of customer acquisition

HR has positive and negative employee engagement and retention rates

CEOs and business leaders get these metrics. They know what good and bad looks like. They know the leading indicators for success. They know what levers they can pull when those department leaders need help.

Engineering is often reduced to, “are we on track to deliver XYZ feature by the deadline?”

Their intentions are good and they want to understand, but most CEOs don’t have an engineering background and therefore they don’t have tools they need to understand engineering in the same way.

Then all of sudden your CEO catches wind that there is a metric called Velocity that your agile software development teams are tracking.

You say… “Each team uses it to estimate how much work they can get done in a sprint.”

They hear… “We can use it to measure dev org output over time and rank team performance.”

They say… “Let’s start presenting velocity in every executive meeting and at every all-hands!”

What you should think… “It’s a trap!”

What can go wrong

At this point in the conversation, our spidey senses should be going crazy. Velocity at the executive table? Do not proceed.

Tips to navigate this situation

This is our time to gracefully push back.

• Ask questions to understand what exactly your executive team is looking for. Is it team performance data? Are they worried about deadline predictability? Are they wondering about the impact of the new hires they invested in?
• Start with yes. “We can absolutely be data-driven.”
  (I had a boss that told me “start with yes”, then get to the outcome you want. That concept has served me well over the years — especially with CEOs 🙂 )
• You know that it will damage culture and encourage sandbagging. But consider keeping all of that to yourself and just explain why Agile Velocity is not accurate when used for anything other than individual team sprint capacity planning. A
• Come with an alternative. Cycle time is a great one because execs already understand sales cycle time. Software development cycle time is the equivalent for engineering teams.

3) Agile Velocity as a false predictor of project delivery date

We’ve gone through two of the basic traps. Don’t use velocity as a performance metric at the executive table and don’t use velocity as a team comparison metric.

Now things get more interesting…

The scene looks like this: A project manager wants to estimate when a project is going to be delivered. Easy, right? A little too easy. We know the team’s velocity over the past few iterations, all we need to do is break the epic down into stories, assign story points, and do math to see how long it will take to be delivered. Done.

Unfortunately, using velocity to estimate the delivery of a substantial long term project spanning numerous sprints does not work. I’ve seen it attempted many times and I’ve never seen it succeed. At best, with good intentions, a lot of false expectations are created. At worst it is a pressure tactic to pin a delivery date onto the engineering team.

Why this never works:

• Capacity planning only has value/accuracy one sprint at a time.
• It takes many iterations for a team to really understand their sprint capacity.
• Estimating the amount of work a story takes increases in accuracy as the project progresses.
• Mapping out stories too many iterations in advance doesn’t work. Too much change.
• There is an inherent pressure to hit a date and estimates get translated into commitments through a game of telephone with business stakeholders.
• Teams start bending story point values to respond to the pressure.

Tips to navigate this situation:

Predicting delivery dates is the golden problem in software engineering. Always a combination of art and science. Mostly never accurate. Using velocity and story points to map out a long term estimation will not only come out inaccurate, it puts the wrong pressure on the estimating developers.

• Propose providing a low accuracy ETA that is not shared publicly
• Update the ETA each iteration as accuracy, predictability and confidence grows
• Break deliveries into small MVPs and communicate progress early and often

The outcome that I have seen most often is that it causes pressure on developers to smudge (inflate) the actual amount of story points per story. Once that starts happening, a vicious cycle begins.

What’s at stake?

When velocity is used in the wrong way, it causes a negative ripple throughout the engineering organization damaging culture. Using velocity as a performance metric at the executive table, to compare team performance or to use for long term project estimation, the following behaviors tend to occur:

• Developers will inflate story point estimates to increase “performance”
• Capacity planning per iteration will become inaccurate
• There will be a culture shift towards new value performance — lowering quality and then delivery speed — loss of investment profile balance.
• Teams will be pitted against each other competitively, not promoting collaboration to deliver together.
• Leading indicators that actually drive team performance improvement will not be measured — performance will not actually increase.
• Developer morale will dip as the emphasis will only be on new value output

Is it ever OK to share agile velocity numbers outside of a single team?

Here’s what Kara has to say about it:

“Velocity can be very helpful outside the scrum team, but it must be used very carefully.

As a positive example, the use I like best beyond the team is when engineering managers and leaders look at the *trend* of velocity over time for their teams. This can be presented by the scrum master as perhaps a rolling 6-month trend for each team. A sudden drop in velocity on a team, or a downward trend over time, should prompt the manager to ask questions and investigate whether there is something with which the team needs her help. For example, a sudden velocity drop could indicate that a team has begun new and unexpectedly challenging work, or has delved into complicated technical debt. In either case, management can often offer useful support.”

But she cautions:

“By contrast, velocity can be destructive when used improperly outside a team. Managers can be tempted to compare velocity across teams and question why one team is “slower” than another (when in truth, velocity is subjectively set by a team and thus relevant only to that team), Or, leaders can be tempted to average velocity across teams to get a sense of “overall” velocity for the organization. Either of these uses of velocity can result in engineering leadership making ineffective or even damaging decisions in a variety of areas including delivery predictions, policy or process decisions, organizational changes, or even reward or promotion choices. A better solution in either of these cases would be to compare commit to complete ratios across teams, which tells you which teams are using velocity well to predict what they can get done, and which are struggling with this skill.”

Alternative metrics

Some of us fell into the trap of using agile velocity in the wrong way because it was there and we didn’t have other metrics to use instead. That’s not the case any more.

Once you know what kind of team culture you want to create and what business outcomes you’re trying to support, there are a number of engineering metrics you can use to measure your progress without damaging trust with your developers.

I created LinearB to help engineering and product leaders use metrics without damaging culture. Click here to see your team’s Cycle Time, Investment Profile and Delivery Risk Index free for 21 days with our self-service trial.

Originally published at https://linearb.io on May 12, 2020.