Measuring Software Engineering Productivity — Part I
Being more productive means we can produce similar outputs with reduced inputs, or produce more outputs with similar inputs, i.e. being more efficient.
A company’s input for its engineering department is the engineering costs (salaries, infrastructure, licenses, etc.). These costs contribute directly to the operating expenses of the company. A company would love to have a more productive workforce, including the software engineers. Unfortunately, often the outputs of engineering does not reflect its productivity.
Engineering Outputs vs Productivity
Software engineers write codes to produce softwares. However, producing more outputs (like LoC, num of PRs/commits/softwares, sprint points) doesn’t always mean that the engineers are more productive; often the opposite is true. Let’s aim for less outputs, then?
Software engineers create softwares to fulfill business ideas, over time; if their engineers can sustainably realize one business idea after another with little to no code, and without additional costs, a company would be very happy (probably, if its competitors couldn’t). Unfortunately, that’s not the reality. Business ideas often translate to many new lines of codes which must not break the existing ones.
So, measuring software engineers productivity by the outputs might be a bad idea, for now. What should we do then? As productivity correlates with efficiency, we can try looking at the engineering efficiency instead. For software engineers, this means the efficiency of the engineering process.
How productive is my engineering?
To answer this question, we need to have reliable data about the engineering. Data can help us:
- Know the current state of our engineering (i.e. the answer of the question)
- Confirm the improvement of investing in engineering initiatives
- See trends (especially deterioration), which should elicit responses
- Benchmark our performance with the industry
Engineers follow a certain process in their day-to-day work. It’s often represented in the “DevOps cycle”, here’s a simplified one:
Design → Build → Learn → Design (→ Build → …)
Writing codes, running tests, releasing, and monitoring are the way software engineers sustainably build something and learn about it.
So, let’s start from measuring the efficiency of these 3 phases. The first thing that comes to mind about efficiency is often the latency, i.e. how long the engineers take to design, to build, and to learn. Lead time for changes is one common metric which tries to capture this of the build and learn phase. Pull request, compile, test, and deployment time are the lower level metrics, which might help us prioritize tech investments better.
When metrics are used to determine incentives, people’s behaviour might change, improving on what they’re measured by. Hence, the metrics need to be chosen carefully, and it’s probably better to use more than 1; 3–5 good metrics should be harder to gamify. More metrics also give us a more complete picture of the world. To achieve this, though, we should measure with different dimensions. In the four golden signals, latency is complemented by traffic (how often, e.g. change frequency), error (ratio, e.g. change failure rate), and saturation (e.g. engineering infrastructure resource utilization).
If software engineers can actualize business ideas faster, more frequent, with high probability of success, they should be able to do their engineering works more productively.
While higher saturation usually means a process is more cost-efficient, it might come with a performance trade-off.
Is that all?
By following the same Design → Build → Learn cycle, we could have a data collection system which captures the efficiency of engineering process. Still, there are many ways to collect, present, and use the data. We also haven’t talked about the people (human) side of engineering, which is also a key component of the actual productivity. So, no, we’re not done yet. Having good data on the engineering process is a good start, but there’s still a long journey ahead.
Inspirations
Books:
- The SRE book — chapter 6,
- Software Engineering at Google — chapter 7
- Developer Productivity Engineering by Gradle
Reports:
- DORA’s State of DevOps research reports
Videos:
