KPIs for Engineering (Managers)
For the impatient, glance at the key takeouts at the bottom. “Listen” button at the top for vision free while hiking 😉
For each KPI, follows: a paragraph about the KPI definition, benefits, what it measures, a paragraph on my experience with it (useful example of how I used it), and a Tags paragraph, so you can search each one in case you need to use it at a certain organizational level. Make sure you look at the benefit in order to understand whether you need to focus on it currently or phase it off for another time.
Starting with perhaps the most prioritized and frequently used¹ metric, Lead Time. Lead Time is the overall time to deliver an increment of software from the appearance of a new customer/PO work request in the workflow to deployment to live (complete SDLC). Measures continuous delivery. High level KPI useful for all engineering management levels, especially C-Suite as it gives insight into how effectively an organization delivers. The main benefit/outcome is increasing the organization’s ability to receive new features (increase ‘velocity’ of the delivery team) by shortening the Lead Time.
During my Pacesetting VPoE leadership, we maintained high level meetings with the SEMs where we discussed the Lead Time and an action plan on its improvement. SEMs were coming up with and driving the plan. It was the VPoE’s responsibility to refocus (by setting goals / OKRs or other) on the most impactful parts of the Lead Time, and enable the SEMs if required. We also used Lead Time in our Iteration Review meetings during SAFe or Scrum of Scrums within enablement companies to understand how the team(s) are doing. From there we analyzed what is offending and created an action plan.
Tags: KPI Granularity = high level; Organizational Level = all; Key Organizational Level = C-suite; Eng Management Level = all; Key Eng Management Level = C-suite, Eng Manager of Managers;
Following up with Cycle Time, also prioritized high but this time by the most granular organizations, the eng team. Cycle Time starts when new work enters “in progress” (someone is working on it), and ends when the work is ready for releasing or after being released. It is a dev metric that measures the efficiency of the dev process, its health. The main benefit/outcome is speeding up the dev process by finding bottlenecks and reducing them.
For example, as an SEM for a Pacesetting company, I found out that it took 20% of the time to do the E2E testing effort. By investigating, I identified that 60% of the time, a. manual testing tool was not used properly (manual testers were spending a lot of time questioning each other on already written manual test scenarios), and 40% of the time b. some of the E2E scenarios were written in a way that was testing part of the service (no proper E2E scenarios). The QA and I had a discussion and identified a few process changes that could be made to first tackle a. as we spent most of the time there. The next few iterations I checked regularly whether the new process is being followed, while paying attention to the QA quality bar — the minimum standards required so the work can be passed to the next phase. You need to make sure that the speed doesn’t lower the quality bar. While QA was finding the rhythm in improving a., we discussed and worked on b. We identified which ‘E2E’ tests are not proper, and checked whether there are tests for those in the appropriate level of the testing pyramid. An integration test, a PACT test here and there, and we reduced b. to near 0. We also doubled down on automating the critical areas, least brittle, manual E2Es. Automation was part of business as usual in the pacesetting company.
Similarly, I have (you could have) identified bottlenecks in other parts of SDLC. Developing tasks in a certain area, or technology, writing UTs, ITs, contract tests. For each, there is a different set of solutions.
Another example, while working in a Enablement Service company, by using Cycle time, in our SAFe process iteration, we identified that the acceptance of tickets (reviewing by the SEM for technical parts or PO for business parts) was slow, and it impacted the overall cycle time since some of the tickets were rejected later in the iteration. Similarly, yet another example, code reviews were lagging and creating the same issue. We agreed on at most 4hrs commit being unreviewed, and a short discussion (pairing) between the implementer and main reviewer on the task in order to discuss on the concept / direction of how it would be implemented. With this we prevented most of the late rejections.
If you haven’t figured it out so far, the better the Cycle Time, the better the Lead Time, as it is a subset. This is why the senior Engineering Managers (DoE, VPoE, C-level) will look at the Lead Time and request a more detailed inspection of the Cycle Time by the Engineering Managers or teams. The more autonomy (or the lesser enablement required by) the team or Engineering Manager shows in solving the Lead Time challenges, the more senior or ripe that team or Eng Manager is, and the more opportunities for growth will be given. The more senior the engineering manager is (highest c-level), the lower amount of time they have to explain how a challenge needs to be solved³⁴⁵⁶.
Tags: KPI Granularity = narrow level; Organizational Level = team; Eng Management Level = SEM; Agile Framework = all;
The third KPI that is usually pretty high (2nd) in the priority of engineering operations¹ is Deployment Frequency. It is the frequency of deploying (the developed features) to live. Measures continuous delivery. It is a High level KPI used by both the (operations/devops) team and the senior eng management levels, as it gives insight into how fast the organization propagates changes to the customer. The main benefits/outcomes are a. the ability to get early feedback and with that, do changes in an isolated manner, rather then deploying a big chunk (epic), and having it dropping the customer use from only a subset of offending changes => spending more time and resources by fixing the dependent code and reducing the availability time of the non-offending code/features for the customers; b. time to market (reaching the customer sooner than your competitors). This is done by organizations through deploying rapidly small software increments to live.
When I entered a pacesetting company as an SEM I was given to manage 4 products (this was a company that grew by acquiring other products/companies). I managed one more product than the other SEMs (my first achievement — something that set me aside within the company). These products had no release for >2 years. By measuring the Deployment Frequency KPI, setting it as part of the product’s OKRs (making it a priority), and hence understanding the challenges (in this case, lack of tests on all levels in the pyramid, lack of automation (of tests and deployment), …) and working on them with the team, I managed to make them releasable each quarter. Once I became a VPoE, I also worked on an initiative called ShipEvery7 days, in order to ripe the benefits a. and b. (check above). I drove the creation of the ShipEvery7 checklist for my organization (SEMs/teams/products). Made the products releasable every 7 days, proving the checklist impacting positively the delivery speed / deployment frequency. Once the SE7 checklist was presented, it was adopted by the whole organization of ~100+ products (another achievement of mine). With time, senior eng management was tasked with assessing which checklist items were most impacting and would take the least amount of time to automate. We automated part of it, and even further increased the delivery speed.
My experience with enablement companies shows that going into higher deployment frequency (DF) depends on different factors. For service companies, it depends on the client. I used to work for a client that had a high engineering maturity (usually after the product has years of tenure), the DF is high. When working with other types of clients, usually it meant that clients were aware of operations’ benefits (proposed by the service company), but the priority went into the development and operations was a side thing. In startup companies, depending on the leadership’s operational impact, they might have pushed for operations as a second priority, but it was all about problem solving, autonomy, usually with little time. A startup I used to work for had the Lead Time, NetPromoterScore prioritized and had the Deployment Frequency right around the corner. So a skeleton (the minimum automated operations) for deployment were there, but there were a lot of manual steps and the edge cases for deployment were part of the knowledge of the developers and operations (devops) employees.
It’s all about priorities, but I can’t stress enough the importance of having at least the minimum automated operations to make quick deployments as a second priority done after/while you are looking at the Lead Time (or Cycle Time).
Tags: KPI Granularity = high level; Key Organizational Level = (Operations) Team, Senior Eng Management; Key Eng Management Level = (Operations) SEM, Eng Manager of Managers;
What follows below is a list of other KPIs ordered by frequency that I (or others) used. Though, you might not be using them in this order. You would be faced with a challenge, and by looking at the benefit from a certain KPI, decide which one is important to you (for your current quarter for example). Some of the KPIs will be part of business as usual, but you need to choose one to three KPIs that will be the ones that will make a big positive impact for your highest eng challenge(s).
Choosing the appropriate metrics depends on:
- the company culture (for example, in a top-down company most of the KPIs and OKRs will be generated and required of you, whereas in a flat hierarchy, you would need to generate these yourself or with your team)
- the phase the product or company is in (in one company, we had a phase where we were doing investigations (RCAs/5whys) intensively as we noticed that the solutions from previous quarters were not reducing defects greatly for instance; than we had a phase where we focused on ship every 7 since we wanted to get faster feedback from clients and time to market; than a phase to focus on reducing costs; a phase to focus on SEMs writing tech deep dive improvements for engineers and VPoEs writing improvements for SEMs. In an enablement company, you might be just starting with the team and it might be crucial for you to create relationships and psychological safety with your team and keep just the business as usual metrics for a period without creating one to three high level focused challenging metrics)
- your role (the more senior eng management role you have, the more it would be expected of you to define the high level metrics; if lower level eng management role, than you need to define narrower metrics; if you are a TL you might be able to propose changes in metrics, but might be that you need to come up with metric improvements during an iteration review ceremony)
- organizational level (metrics might be required to be generated by (group of) groups or even on an individual level)
- other
, so make sure you understand the environment and conditions you are in, and try to find a creative solution within the limits of that environment. Don’t make the mistake of rushing to change the limits immediately. First, understand and learn and practice within the limits for a period, and then and only then, think about challenging the limits. You might think that you see an opportunity, but might be mistaken or the team/organization might not be ready for that change, so make sure you are creative within the limits.
4. Time to restore service
5. Sprint/iteration Velocity or Delivered Story Points
6. Sprint Burndown
7. Release Burndown
8. Flow Efficiency
9. Escaped defects
Defect Containment (Reduce the trailing 4 weeks average incoming defect rate from Qn(QarterN) baseline of <x> by <y>) was one of our high level KPIs in order to improve on the quality of the product.
10. Code Coverage
11. Sprint Target Completion
12. Cumulative flow
13. Code Stability or Code Churn
14. Change fail rate
15. Code Simplicity
Different tools like SonarQube have some of the above narrower engineering KPIs cover and might give even more narrower KPIs that you can use depending on your challenge.
Quality metrics counterweight for non-quality metrics
If you dive deep into bettering your performance non-quality metrics, you might end up reducing the quality of your output. So counterweight your non-quality metrics (like Cycle Time, Deployment Frequency…) with your quality metrics (like Escaped Defects), Definition of Done, Definition of Ready, and any other Quality Bars existing between different type of teams within your organization (we used to have a Quality Bar for a architectural deliverable produced by the Architecture Teams that was required by the Feature teams — we wouldn’t pass a deliverable if it didn’t pass the QB) or between the organization and the customer (Net Promoter Score).
A few words about agile methodology metrics
All agile practices Kanban, Scrum, Scrum of Scrums, SAFe… rely on some set of metrics.
The bare minimum for Kanban, for instance, is Velocity. For high level metrics (like OKRs, that I am going to be discussed in another post — in the oven), Scrum has a Sprint Goal Success, SAFe, for instance, has iteration Objective and iteration Key Results (OKRs) and even further it has an OKR for the whole PI (Product Increment — usually lasting for a quarter).
It might be the case that you are working under some less restrictive agile process like Scrum where few metrics are recommended. In that case, if you require, you will go into defining your own KPIs and OKRs. On the other hand, you might be working within a more structured agile methodology, like SAFe, and then a lot of the metrics will be recommended. For instance, SAFe has Value Stream KPIs like Retention (Software Product), Net Promoter Score (Fulfillment), Mean Time to Resolution (Support), has its OKR structure (as described above) and Flow metrics (for instance Flow Time, that is equivalent to the Lead Time). For more, check also the SAFe “Four Critical Success Factors for Effective Measurement”.
Key takeouts
How do you choose the metrics? You would be faced with a challenge, and by looking at the benefit from a certain KPI, decide which one is important to you (for your current quarter for example). Good start to choose from is the ordered list I show in the post, check any tools for narrower KPIs (like SonarQube) and check the recommendations from the agile methodology you follow (for instance SAFe). Some of the KPIs will be part of business as usual, but you need to choose one to three KPIs that will be the ones that will make a big positive impact for your highest eng challenge(s).
Choosing the appropriate metrics depends on:
- the company culture
- the phase the product or company is in
- your role
- organizational level
- other
, so make sure you understand the environment and conditions you are in, and try to find a creative solution within the limits of that environment. Don’t make the mistake of rushing to change the limits immediately. First, understand and learn and practice within the limits for a period, and then and only then, think about challenging the limits. You might think that you see an opportunity, but might be mistaken or the team/organization might not be ready for that change, so make sure you are creative within the limits.
Counterweight your non-quality metrics (like Cycle Time, Deployment Frequency…) with your quality metrics (like Escaped Defects), Definition of Done, Definition of Ready, and any other Quality Bars existing between different type of teams within your organization (we used to have a Quality Bar for a architectural deliverable produced by the Architecture Teams that was required by the Feature teams — we wouldn’t pass a deliverable if it didn’t pass the QB) or between the organization and the customer (Net Promoter Score).
Check above for the ordered by frequency (of need) 15 KPIs that helped me in my career.
Looking forward to hearing your questions and comments on experiences. 😉 Feel free to do it in public or in private.
Resources
The “KPIs for Engineering (Managers)” Linkedin resource for additional audience to comment, interact and share with.
Metrics for Engineering (Managers) (in the oven ..)
OKRs for Engineering (Managers) (in the oven ..)
SAFe Metrics
Scrum Metrics 1 2
Other:
General One on ones (for all seniority levels of managers):
Cadence (1–1s)
Agenda (1–1s)
First 1–1s
Prior to 1–1s (in the oven ..)
During One-on-ones
After 1–1s (in the oven ..)
Location (1–1s) (in the oven ..)
One-on-Ones for (Engineering) Manager of Managers:
Cadence (MoMs’ 1–1s) (in the oven ..)
With whom? (MoMs’ 1–1s) (in the oven ..)
Agenda (MoMs’ 1–1s) (in the oven ..)
During a 1–1 (MoMs’ 1–1s) (in the oven ..)
During Interviewing (MoMs’ 1–1s) (in the oven ..)
1–1s with Senior Peers from other functions (MoMs’ 1–1s) (in the oven ..)
Reference
My experience. :)
[1] Nicole Forsgren PhD, Jez Humble, Gene Kim , “Accelerate” book (forward by Martin Fowler)
[2]
[3] Camille Fournier, “The Manager’s path”, 2017, book
[4] Will Larson, “An Elegant Puzzle: Systems of Engineering Management”, 2019, book
[5] Dr. James Stanier, “Become an Effective Software Engineering Manager: How to Be the Leader Your Development Team Needs”, 2020, book
[6] Marcus F., https://lnkd.in/dghjKmQB, youtube videos
