Agile Metrics — The Good, the Bad, and the Ugly

Stefan Wolpers
Dec 11, 2016 · 7 min read

TL;DR: Agile Metrics

To address the team level, qualitative agile metrics typically work better than quantitative metrics. At the organizational level this is reversed: quantitative agile metrics provide better insights than qualitative ones.

Good Agile Metrics

A metric should therefore be a leading indicator for a pattern change, providing an opportunity to analyze the cause in time.The following three general rules for agile metrics have proven to be useful:

  1. The first rule of tracking meaningful metrics is only to track those that apply to the team. Ignore those that measure the individual.
  2. The second rule of tracking metrics is not to measure parameters just because they are easy to follow. This practices often is a consequence of using various agile tools that offer out-of-the-box reports.
  3. The third rule of tracking metrics is to record context as well. Data without context, for example, the number of the available team member, or the intensity of incidents during a sprint, maybe turn out to be nothing more than noise.

For example, if the (average) sentiment on the technical debt metric (see below) is slowly but steadily decreasing, it may indicate that the team:

  • May have started sacrificing code quality to meet deadlines or
  • May have deliberately built some temporary solution to speed up experimentation.

While the latter probably is a good thing, the first interpretation is worrying. (You would need to analyze this with the team during a retrospective.)

Good Qualitative Agile Metrics: Self-Assessment Tests

All you have to do, is to run the questionnaire every four to six weeks during a retrospective, record the results, and aggregate them:

Image for post
Image for post

In this example, we were using a kind of estimation poker to answer each question with one of the three values green, orange, and red. The colors were coded as follows:

  • Green: It worked for the team.
  • Orange: It sort of worked for the team but there was room for improvement.
  • Red: It either didn’t apply, for example the team wasn’t using burn down charts, or the practice was still failing.

If the resulting Scrum practices map is getting greener over time, the team is on the right track. Otherwise, you have to dig deeper to understand the reasons why there is no continuous improvement, and adapt accordingly.

In addition to this exercise, I also like to run an anonymous poll at the end of every 2-week-sprint. The poll is comprising of three questions that are each answered on a scale from 1 to 10:

  1. What value did the team deliver last sprint? (1: we didn’t deliver any value, 10: we delivered the maximum value possible.)
  2. How has the level of technical debt developed during the last sprint? (1: rewrite the application from scratch, 10: there is no technical debt.)
  3. Are you happy working with your teammates? (1: I am looking for a new job, 10: I cannot wait to get back to the office on Monday mornings.)

The poll takes less than 30 seconds of each team member’s time, and the results are of course available to everyone. Again, tracking the development of the three qualitative metrics provides insight into trends that otherwise might go unnoticed.

Please hit the “heart button” 💚 below, if you found this post useful–it would mean a lot to me!

Good Quantitative Agile Metrics: Lead Time and Cycle Time

In the long run, this will not only require to restructure the organization from functional silos to more or less cross-functional teams, where applicable. It will also require to analyze the system itself, for example, to figure out where value creation is impeded by queues.

To effectively identify the existing queues in the product delivery process, you start recording five dates:

  1. The date when a previously validated idea, for example a user story for a new feature, becomes a product backlog item.
  2. The date when this product backlog item becomes a sprint backlog item.
  3. The date when development starts on this sprint backlog item.
  4. The date when the sprint backlog item meets the team’s ‘Definition of Done’.
  5. The date when the sprint backlog item is released to customers.
Image for post
Image for post

The lead time is the time elapsed between first and the fifth date, the cycle time the time elapsed between third and the fourth date.

The objective is to reduce both lead time and cycle time to improve the organization’s capability to deliver value to customers. This is accomplished by eliminating dependencies and hand-overs between teams within the product delivery process.

Helpful practices in this respect are:

  • Creating cross-functional and co-located teams
  • Having feature teams instead of component teams
  • Furthering a whole-product perspective, and systems thinking among all team members.

Measuring lead time and cycle time does not require a fancy agile tool or business intelligence software. A simple spreadsheet will do, if all teams stick to a simple rule: note the date once you move a ticket. This even works with index cards.

Other Good Agile Metrics

Bad Agile Metrics

Some of the many factors that make even intra-team sprint comparisons so difficult are:

  • The team onboards new members,
  • Veteran team members are leaving,
  • Seniority levels of team members change,
  • The team is working in unchartered territory,
  • The team is working on legacy code,
  • The team is running into unexpected technical debt,
  • Holiday & sick leave reduce capacity during the sprint,
  • The team had to deal with serious bugs.

Actually, you would need to normalize a team’s performance each sprint to derive a value of at least some comparable value. (Which usually is not done.)

Additionally, velocity is a metric that can be easily manipulated. I usually include an exercise in how to cook the “agile books” when coaching new teams. And I have never worked with a team that did not manage to come up with suitable ideas how make to sure that it would meet any reporting requirements based on its velocity. You should not be surprised by this — it is referred to as the Hawthorne effect:

The Hawthorne effect (also referred to as the observer effect) is a type of reactivity in which individuals modify or improve an aspect of their behavior in response to their awareness of being observed.

To make things worse, you cannot compare velocities between different teams since all of them are estimating differently. Which is totally fine as estimates are not pursued for reporting purposes. They are a side-effect of the attempt to create a shared understanding among team members on the why, how, and what of a user story.

So, don’t use velocity as an agile metric.

Read more on velocity here: Scrum: The Obsession with Commitment Matching Velocity.

Ugly Agile Metrics

Equally useless “agile metrics” are, for example, the number of certified team members, or number of team members that accomplished workshops on agile practices.

Conclusion

What are you measuring to track your progress? Please share with us in the comments, or join our Slack team “Hands-on Agile” — we have a channel for agile metrics.

Related Posts

Product Backlog Refinement — Agile Transition Part 2

How to Build Offline Boards — Agile Transition (Part 3)

Please hit the “heart button” 💚 below, if you found this post useful–it would mean a lot to me!

Do you want to read more like this? Well:

The Startup

Medium's largest active publication, followed by +684K people. Follow to join our community.

Sign up for Top Stories

By The Startup

A newsletter that delivers The Startup's most popular stories to your inbox once a month. Take a look

By signing up, you will create a Medium account if you don’t already have one. Review our Privacy Policy for more information about our privacy practices.

Check your inbox
Medium sent you an email at to complete your subscription.

Stefan Wolpers

Written by

I have worked for 14-plus years as an Scrum Master, Product Owner, and agile coach. Professional Scrum Trainer (PST) with Scrum.org.

The Startup

Medium's largest active publication, followed by +684K people. Follow to join our community.

Stefan Wolpers

Written by

I have worked for 14-plus years as an Scrum Master, Product Owner, and agile coach. Professional Scrum Trainer (PST) with Scrum.org.

The Startup

Medium's largest active publication, followed by +684K people. Follow to join our community.

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight.

Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox.

Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month.

Get the Medium app