Agile Metrics — What Matters and Why?
Over the last couple of decades, agile has taken hold of the software development industry. Every technology organization is either agile or in the process of transformation into an agile organization — it is common to call them ‘Wagile!’ The traditional project management metrics like productivity, schedule variance, cost variance etc, may not be useful in an agile system. In fact, they can be counterproductive.
There is a plethora of metrics that are meant for agile systems as well. The question is — which metrics among them do really matter? Here is a practical take on it. A selected subset of agile metrics that are useful within most contexts and organizations.
For the sake of simplicity, let’s classify the metrics under the following three categories — Productivity, Stability, and Quality, as follows.
Productivity
The following are the key metrics regarding the eficient use of resources to deliver certain outcomes.
OKR (Goal) Contribution
It is important to measure how much of your team's effort is being put towards accomplishing the strategic goals of the organization or the business unit. For example, it could show that, on average, 60% of the team’s capacity goes towards OKRs in each sprint. 50% or more of the effort should ideally go towards the OKRs or strategic goals of the company.
Value Delivered
This metric helps measure the value delivered with each sprint. It is easier in product led organizations, where the prioritization formula or matrix involves the estimation of value in the same way that the development teams estimate story points for effort. For instance, WSJF (Weighted Short Job First) prioritization method estimates value points in terms of business, risk reduction, time-criticality and opportunity enablement. RICE method estimates value points across reach and impact.
Burndown
Burndown charts are an easy way to visualize the progress made overtime to complete a defined set of tasks. The X-axis represents the set of tasks for a timebox and the Y-axis denotes the incremental units of time within the timebox. Burndown charts are used to monitor the completion of tasks overtime for entire releases, epics and sprints. Sprint burndowns are highly recommended. It helps to keep the team on track on a daily basis during a sprint.
Velocity
In broad terms, agile velocity represents the story points completed in each sprint. This helps to estimate the number of sprints it takes to complete a body of work estimated in story points. There are many variants of velocity — maximum velocity, average velocity, and optimal velocity. It is best to choose optimal velocity — the average number of story points it takes per sprint to deliver a body of work according to the acceptance criteria, without any open defects.
Lead Time
Lead-time denotes the time it takes from the creation of a work item (Story, Feature or Epic) to its completion. This is a good measure of efficiency of the team. The lower the lead time, the more efficient the team is in terms of turning ideas or outstanding requests to working software.
Cycle Time
Cycle-time denotes the time it takes from the start of work on a work item (Story, Feature or Epic) to its completion. This is a good measure of efficiency of the team’s development, testing and release process. The lower the cycle time, the more efficient the team’s development, testing, and release process is.
Cumulative Flow Diagram (CFD)
The Cumulative Flow Diagram gives you a snapshot of your entire intake, development, testing and release process from start to finish. It is a simple visualization that lets you identify bottlenecks instantly.
Net Promoter Score
Net Promoter Score is a percentage score of the difference between your promoters and detractors. The customers are asked to give a score from one to ten as an answer to the question — how likely are they to recommend the product to others? The ones who scored 9 and 10 are promoters, 8 and 9 are passives, and 6 and under are detractors. More info here. NPS is arguably the most important metric of them all, since the customer is the best stakeholder to determine whether the team has delivered anything of value to them.
Stability
The following are the metrics that will help the team to ensure that the process of delivery, including the development, testing, and deployment runs in a stable, predictable and sustainable manner.
Planned to Done Ratio
The Planned to Done ratio represents the percentage of tasks or work items completed as planned at the end of a timebox. This gives an idea of ability of the team to estimate the tasks correctly, resolve impediments, skill gaps, and of course to deliver in time. Aim for more than 80%. Paying close attention to this metric will ensure higher predictability in the long run.
Work In Progress (WIP)
WIP is a key metric from Kanban — you can find it in Scrumban too. WIP is the number of work items that are actively being worked on. The aim is to limit the work in progress, so that the context switching is less, and that the team resolves blockers in time. Ideally, we should aim for 1–2 tasks per scrum team member at a time.
On-time Release Rate
This number is the percentage of releases that are completed on the release date as planned. This number signifies the predictability in terms of releases done on time. The higher the number, the lower the need to replan and reorganize — which causes a significant waste of the team’s capacity.
Failed Deployments
This number shows the number of deployments that have failed in a given timeframe. It is used for production deployments and for the deployments to the lower environments. It is a measure of code stability and shows whether the development team is actually ready with potentially shippable code at the end of a sprint. It is a measure of stability of the environments as well.
Team Turnover Rate
This is one of the most important metrics, when it comes to stability. The turnover rate represents the rate at which the agile team members leave the team and are replaced. Turn over are costly - hiring cost, ramp up time, and the loss of institutional knowledge - to name a few. Some amount of churn is inevitable. But if it is more than the industry norm, the team should address it. In general, the turnover rates for the tech industry are between 10 to 15% per year. The top three reasons for churn are compensation, career advancement opportunities, and work conditions. So you already know what to do if your turnover rate is high.
Happiness Score
Similar to CSAT (Customer Satisfaction Score), this is a scoring done usually on a five point scale on the mean happiness of the team. The survey is usually done as part of sprint retrospective and tracked over time to monitor the deviation sprint after sprint. Happiness score has a direct correlation to the Turnover Rate. It is inversely proportional to the Turnover Rate — the more the Happiness Score, the less the Turnover!
Quality
Given below are some of the key metrics that will help the team ensure that the shipped code is of acceptable quality.
Defect Resolution Time
Defect Resolution Time (DRT) or the Mean Time To Resolution (MTTR) is the average time it takes for the development team to resolve a defect. Shorter the better. Mean DRT or MTTR is usually tracked for defects across all environments. The SLAs for DRT will vary across organizations and teams. In general, the teams should aim for a maximum DRT of 1 day for Critical, 1–2 days for High, 5–7 days for Medium and 10–14 days for Low defects. MTTR is inversely correlated to customer satisfaction and hence, important.
Code Coverage
Code Coverage is the percentage of the lines of code covered by unit tests. Higher the better. Aim for coverage greater than 80%. Code coverage numbers can be run for every build. This ensures that the team is not deploying code that is not ready. This doesn’t cover the other kinds of testing. Hence, high code coverage doesn’t necessarily mean that the code is of high quality.
Test Coverage
Sometimes the term Test Coverage is used interchangeably with Code Coverage. But they are not the same. While code coverage measures whether the written code is covered by tests or not, the Test Coverage measures whether the functional requirements are covered by the existing set of test cases. Aim for 70–80% coverage. The subcomponents are requirement coverage, product coverage, risk coverage and boundary coverage - in case you need more granularity. This metric ensures that the functionality meets the requirements and only the requirements.
Escaped Defects
This is the number of defects identified after production deployment. It is a crude measure of software quality. This number should ideally be zero. If the number is not zero, the team should reassess its QA and deployment processes to bring it down to zero.
The later the defect is identified in the development cycle, the higher the costs to fix. Hence, this is one of the critical metrics that the agile teams should track.
Bringing it all together
Of course, the agile way is the time tested way to ship software that maximizes the value to customer. There are dozens, if not hundreds of metrics and opinions around the process. If we treat all the metrics as important, then no metric is really important. The key is to choose the right set of core metrics that fits your team. Needless to say, idea is not worth anything without execution. Yes, it is easier said than done. It will be hard at the beginning. Painful and annoying some other times. But once the team matures and knows how to adhere to the basic processes and to track key metrics, the results will be more than worth it.
