A recurring observation I have had over the last 15 years has been of the misappropriate use of Velocity as a measure of team productivity. The most glaring error is focusing on this number as something that should be managed and improved upon. A metric to measure productivity across changes in approach, tooling and/or team constituency is essential. The choice should be Cycle Time not Velocity. Cycle Time which measures the amount of time from when an engineer starts working on a bit of code through to its release to production — the duration from start to the delivery of value to consumers. However any single number alone is not enough. We need a small handful to assure that everything is held in proper context .
The core development unit in many agile implementations is called a story. The team estimates each story, giving it a number of “points” that represent the anticipated amount of work to complete that story. This includes both functional (features) and nonfunctional work (such as testing and performance tuning). Scrum (one of the more popular agile implementations) moves in time-boxed increments called sprints. Velocity is the amount of story-points we have successfully accomplished in a sprint. It is best evened out over a period of time through an average of some sort (mean or median).
It is not uncommon for immature teams to see an increase in velocity as they become comfortable with scrum, their problems domain and their teammates. Mature teams will likely see a small sprint over sprint variance in velocity, however a significant change should not occur (this is true when maturity implies an optimized process). Because of a lack of significant variance, velocity can be serve dual purpose when looking at “amount of work”. First it allows us to reflect on what has been accomplished. Secondly (and more importantly) it serves as an indicator as to what to expect in upcoming sprints.
What changes velocity
We should only anticipate small variances in velocity, but we may witness larger swings caused by these types of changes:
- Change team constituency (positively correlated): As we add resources we should do more work
- Change sprint duration (positively correlated): As we increase the duration of the time-box we should do more work
- Change an approach to quality (inversly correlated)
The last bullet is of concern, quality needs to be first class citizen of the engineering ecosystem. Its well documented that a high quality line of code costs 67% less than a low quality line of code over the lifetime of that code base (http://www.agile-doctor.com/tag/total-cost-of-ownership/). In addition high quality code, with proper automated test coverage, allows the teams to move faster in future sprints and leads to fewer escaped defects.
If we do see a change in velocity, not caused by changing the team embers or sprint duration, this is a cause for concern. Measuring and tracking velocity is a good technique to help you estimate and right size your sprint commitments. It is a useful tool as an indicator of a change in approach (such as quality), but there is little to little to no value in presenting this number outside of the team context. Further we should never compare velocity across teams, as constituency, the work represented by “a point”, sprint duration and the tasks at hand changes significantly between teams.
On Cycle Time
Cycle time starts the moment an engineer picks up a story and completes the moment it is released to production. This is a measure which is worthy of management. In this instance “management” is the tracking and discussion of how a team might reduce average cycle time. A mandate to reduce cycle time at all costs not the intended consequence. Cycle time is fairly easy to reduce:
- Work on smaller stories
- Release more often
By breaking stories into the smallest possible increments and releasing as soon as the work is complete (and signed off) the team is delivering features to its end users as fast as possible.
Impact on Velocity
Changes in cycle time should not impactful change to velocity. Velocity is the the amount of work expected to be completed in a sprint (unit of time). Cycle Time doesn’t speak to the amount of work, it speaks to how the work is accomplished and released. There may be some minor variation in velocity, particularly as teams adapt to managing cycle time (more on the impact of Cycle Time), but the velocity should normalize prior to the point cycle time minimizes. If there is a significant increase in velocity coupled with a decrease in cycle time you might be observing the effects of Goodhart’s Law (https://en.wikipedia.org/wiki/Goodhart%27s_law):
When a measure becomes a target, it ceases to be a good measure.
Avoiding the effect of Goodhart’s Law
No single metric by itself tells the story by itself — increases may be coming at the expense of tradeoffs in other places. In the case of cycle time we want to continue to deliver a high quality product — so we could easily add three metrics to measure quality:
- Escaped Defect Rate
- Unit Test Coverage
- Cyclomatic Complexity
Escaped defect rate help assure that the product your customers are using is not being impacted by poor quality. As a rate stat it combines the cumulative escaped defects per some unit of work. You could phrase it over any unit of work as long as you are consistent. Two units of work that make sense are “per sprint” or “per story” — where the latter makes more sense if releases are more frequent.
Unit test coverage, the percentage of code that is covered by automated unit tests, is seen as a predictor of quality. The higher the unit test coverage the less likely you are to be caught by a surprise defect. The caution is that 100% coverage doesn’t mean that there will be zero defects (tests can be written poorly — bad logic and.or bad assumptions).
Cyclomatic complexity is a measure of the maintainability of a code base. It is essentially the number of lines divided by the number of branches (counted by counting IF, CASE and MATCH statements). The metric has correlated positively to teams ability to iterate quickly on defects minimizing their user impact.
Impact of Cycle Time
Optimizing Cycle time requires teams to invest in automation. The investment in automation is a short term drag on feature development, but once delivered presents an opportunity for continuous deployment (the release of code to production with no manual steps). This requires a high degree oftrust through automated tests and appropriate constraints and visibility into predictive quality metrics (e.g. test coverage, cylcomatic complexity, et al).
A dashboard that covered these five metrics (Cycle Time, Velocity, Escaped Defect Rate, Cyclomatic Complexity and Unit Test Coverage) would be an great start to a software metric program. It helps provide the confidence and establish the trust that the team is performing a consistent and healthy amount of quality work and placing it in the hands of their users as fast as possible.
As an engineer I found software metrics programs liberating. They provided the context, to both effort and risk, that previously lacked visibility. They are able to then cross reference the teams performance with the evolving industry research. Having a confirming outside opinion and visibility into the metrics helps develop trust and a healthier team dynamic.
Beyond Cycle Time
As engineers and engineering leaders we need to become comfortable reporting on our efficiency and progress. Visibility is the start — however deriving the effect variation on different variables have on live cycle profits is probably where you want to take this…