Making Sense of Engineering Metrics

It has always been a challenge to measure the success of engineering teams.

Firstly, because there are so many different types of engineers, software developers, data scientists, etc. that there is no real source of truth for measuring them. You might have a data scientist who doesn’t commit code very often, or you might have a front-end engineer who would make small, but frequent commits. Then again, depending on the project or company they are working for, this thought process could be completely different!

Secondly, your KPI measurements will vary widely depending on someone’s seniority which makes comparing engineers on one time like comparing apples and oranges. A junior developer will spend most of their time committing code, but they should also be spending time reading everyone else’s to learn and develop. By contrast, a senior engineer is often dedicating time to mentoring their team and reviewing others work, so you can’t only analyze their code output.

All this to say, measuring KPIs when it comes to your engineering team is very complex. One thing we can say for certain, is that it’s definitely not about the number of lines of code someone commits. However, despite it being complex, it is important to find some metrics that work for your team and your engineers. Effective measuring will allow you, as a manager or leader, to better coach and support your team, which should hopefully lead to an increase in productivity.

Here are some of the metrics that are becoming increasingly important to understand and track.

Code Churn

Code churn is simply code that is rewritten or deleted after being written. It’s a normal part of the software engineering process because software engineers often test and explore various solutions to a specific problem.

Code churn will always differ between different teams and the type of projects they’re working on. And although code churn in itself isn’t a bad thing, it’s important that it stays consistent. This means if the code churn moves significantly above or below what your team normally achieves, it can signal a problem.

Apart from this, it also gives you an idea of whether a developer is doing quality work. In other words, the more changes they make, the more their code will be prone to errors. Ideally, you want to see code that isn’t full of errors and therefore doesn’t need to be corrected that often.

It can also give you an idea of how the developer works. For example, when we see low churn and low throughput we can assume that the developer is probably a perfectionist. In contrast, high throughput and high code churn indicates to us that there could be a potential problem — the deadline might be too tight, there’s a bad bug, or perhaps incorrect requirements. At Gsquad, we tend to look for developers with high throughput and low churn, although some might label this a unicorn!

Commit-to-Deploy Time (CDT)

This metric tells you the time it takes for your code to go from commit to deploy. Between committing code and the deployment thereof, it goes through testing, quality assurance, and staging — depending on your business. The goal of using this metric is to understand how long it takes code to go into production and what hurdles it’s encountering along the way.

Ideally, if you’re using continuous integration best practices and you have sufficient testing coverage, your code can be deployed in mere minutes. If, however, you rely largely on a manual quality assurance process, your CDT will be longer. This likely means that you have room to improve.

Fast-moving organizations deploy many times a day. In contrast smaller businesses may only see daily deployments. It’s important to have an understanding of what works for your size of business and team and ensure that you maintain that consistency.

Pull Requests

A pull request or merge request is simply a request by a developer to make a change into the code base of the software. This means that you’re pulling changes from another branch of the code and merging the changes with your existing code.

These are often used when there’s a change in a feature of your product. Ideally, you don’t want a request to be open for too long and you want to have consistent merging times. Ideally, you have them open and reviewed within 24 hours, otherwise the entire team could be slowed down. If it takes longer than 72 hours, it could mean that your team is struggling.

Another method that ties into this is merge request rate. This is simply calculated by dividing the total number of merge or pull requests for a month by the number of team members who worked on the project for that month.

At Gsquad, we look for a higher merge request rate because this means that developers are breaking down work into smaller merge requests. These smaller merge requests have a faster review time and get merged quicker. Ultimately, this means we can deliver features faster to customers.

Size and Frequency of Commits

This metric helps you to understand the volume and quality of the work your developers are doing. As mentioned above, we prefer smaller pieces of work committed more often than one big commit that needs to be revised. This is because we want to see incremental small changes over time instead of big changes every two weeks.

You should keep in mind, though, that the size and frequency of commits do not necessarily correlate with productivity. At best, these commits show you the activity of your developers. It’s also important to remember that the frequency of commits and the size of the commits will be different between developers and projects.

You can, however, track whether the frequency of your commits stays consistent over time. This will represent productivity more than just looking at commit frequency and size.

The Bottom Line

While it can be tough to measure and track your engineers, it’s important to do and should be approached as a means of supporting your team, rather than micromanaging. When you have an understanding of whether or not your team is doing well, you can put measures in place to remove blockers, get the support they need, and ultimately succeed.

Stay tuned as we’re working on a blog post about the Gsquad Engineering KPI philosophy and methodology to be published in the future.

We’re also always on the hunt for senior technical talent to join our global network of freelancers. If you’re interested in joining us, find out more here.

Learn more about how the Gsquad model Nurtures our Global Community of Freelancers.