Packlink
according to the 4 Keys metrics
What are the 4 key metrics
The 4 key metrics are a set of (quickly becoming) de facto standard metrics for understanding where a given team or whole department (in our case) sits in comparison with “the rest of the software development world”.
It was first described in the State of DevOps report, first published in 2014. Every year a new report is published, the last being the 2019 edition State of DevOps (pdf) report.
I encourage you to read the book Accelerate that fully describes the scientific methods and statistics used to craft the report.
Why these metrics?
From time to now we have been looking into ways of having visibility on how well the team has been performing now that we were growing.
What we are looking to answer
Growth impact in the delivery
Growing has a big impact (the bigger the smaller you are, like Packlink) in how teams perform because there is disruption in many forms: new colleagues, interviews, technical reviews, etc.
That’s why you have to assume some degree of impact in either quality or frequency.
Our approach has been to invest time in technical reviews and in the interviews so the last part of the process (onboarding) would be the least disruptive.
With these metrics we aim to track this assumption.
Visibility and Health
A key difference with other type of work is how hard is to have meaningful metrics from development to share with other areas of a company like Customer Satisfaction (time to resolution, client satisfaction) or Sales (onboarded clients, churn, etc) in order to give visibility and health of the development area.
These metrics are meaningful in that they show understandable and both business (% of catastrophic errors, time to fix them) and client-oriented ones (how long we take release, to how often we update our product).
Since forever giving proper and meaningful visibility with KPIs other than those of features released or stories completed to non-tech people has been a challenge.
I had read the Accelerate book long ago and found that these metrics could help me to have a closer language to that of those non-tech people that represent the rest of the company at the management level; as well as a high level view of how the team is doing.
Gathering the data
And for obtaining the figures of course you need to have the data to work with.
Deploys
I got the figures from Jira, it took time to be a reliable figure as we were undergoing a big change in our workflows.
Lead time
For the lead time, I relied on an interesting workshop run by our friends from ThoughtWorks España (Spain) with whom we work and that crafted with us the value chain for our “from PR to production” time.
Fail rate & time to restore
In this case, I took the time to register every rollback that was causing a major disruption in our applications.
While we have other kinds of “broken in production” urgent incidents, these vary a lot in nature and can’t be counted without taking a look into them.
I’m considering triaging them for the next iterations.
The road to “production”
During 2019 I did a first review of how they would look like. The process was quite manual and with a lot of manual data processing to have clear figures.
At that moment we had no lead time, time to restore was a gut feeling and change fail rate data quality was improvable.
Looking ahead is automating data gathering and even the report itself.
Packlink in the light of the 4 metrics
Based on them I have created a first report of Packlink tech teams as a whole, the results are quite encouraging.
*Special thanks to the ThoughtWorks team that created with us our first Lead time value chain to explore where we had potential improvements.
** For now we are counting “rollbacks”, that is, new versions that for whatever reason render our applications unstable and have to be rolled back to the existing (previous) version.
Conclusions
With a 16% increase in team size during Q1, we have achieved a 160% increase in releases while keeping the fail rate very low (3%) from a 40% increase in incidents (2 incidents).
These figures give us confidence in our selection and onboarding processes (that we improve at every opportunity), ensuring minimal development disruption and great performance of the different teams.
Lead time remains a key point to address, with Packlink currently in the high performing group and eager to make progress to reach the next level.
Least but not last, if you have made it here I’m sure you would like & enjoy working with and helping us to continue improving. You can go to jobs.packlink.com and take a look!
