AI digital art image by DALL-E 2

Optimizing for efficiency in SWE

How to build high performance software development teams

The purpose of writing for me is often to collect my thoughts and develop them into insights that are very precise and accurate. Today I thought about agile software development, its role in increasing the efficiency and useful output, and a better understanding of how to drive efficiency further. Then I also asked myself questions about low and high performers and how their differences can be explained.

I try o reduce this understanding into high-level principles in this article, just like I did in my airflow documentation on the topic of ETL.

Agile, Scrum and “development process”

Engineers often share with me how they develop software using Scrum. Most of them recount the ceremonies right, but at the same time miss the essence of agile.

Understanding why you use a method rather than following it as established best practice makes the difference between a team that can optimize towards a goals and teams that get stuck.

A course I took was remarkably effective in encapsulating Agile’s purpose in two words:

Eliminate waste

Those two words do not tell you how to do that. So the question one has to ask for every ceremony and process you follow in the Scrum method is:

“Does this process contribute to eliminating waste?”

And from that question, we have to ask another question.

“What are sources of waste?”

Is it possible that by introducing a process, we introduce waste rather than solve it? Any kind of process is probably more likely to introduce bureaucracy and therefore waste than solve it. The best way to solve bureaucracy is automation.

Sources of waste

Not eliminating useless recurring meetings

I worked at an organization that had dedicated scrum masters. It’s great to work with experts in the field, but at the same time there is a tendency to optimize towards enriching the process, tinker with meetings, experiment with different ceremonies and so on.

It probably leads to strong team building and social engagement. But the outcome of their work in the long run depends on what metrics and observations they really use to drive their work. If this is only based on observations of velocity, then it is probably really bad. I will explain that later.

How do you determine if a meeting is useful?

• Periodically ask people around the team, in their syncs, or in group meetings, which meetings they’d like to scrap and for what reason.
• Evaluate if the meeting establishes action points that are followed up. Either way, if people discuss pain in meetings, that could make them happier about it. If action points do not happen, the meeting was useless.

You only build efficiency through action, not by talking about it. If you think you need to talk about it more, reduce scope and start smaller.

• Should the meeting be with the entire team, or a fraction of the team only?
• Avoid designing a very complex system up-front. Think in terms of layers and boundaries of abstraction. A lot of conscientious people like to design the whole system before anyone starts coding, but this leads to premature design conflicts at the wrong moment in time, when there’s not enough context and real life observations available to establish clear arguments on the tradeoffs one has to make.

I heard one company had a drastic policy to delete all scheduled meetings every quarter. Only the useful meetings survived.

You don’t need story points

I’ve worked the past three years without story points whatsoever and life has been great. The times were things were less great; it was because of a lack of control over the scope of development and clearly defined business goals.

Isn’t that what the story points are supposed to solve?

Yes. When I first thought about: “Why am I assigning these numbers to these things?”, it became clear that it is a method to explore story complexity and to get everyone on the same page as to what is being built. But when you approach the problem like that, we also have to consider that we’re in a meeting room, nobody will really look into what’s really required, and we end up in a stalemate of opinions. If you get into a stalemate, create a story to gain information, then build your arguments on your observations and make a proper decision. That avoids compromising with the person with the biggest mouth and opinion. Have bias towards action, not bias towards discussion.

The other problem with story points is that they get referenced in performance evaluations, and carry weight. The planning session should be 100% focused on the stakeholder and your capability to deliver.

Successful project execution is all about:

• Understand the business problem you’re trying to solve.
• Understand customer expectations and constraints.
• Technical perspective: Not leaving huge room to become distracted by premature optimization, technical elegance or efficiency. Only focus on improving what is needed to be improved at that time. Create the milestones short enough to maintain strict focus on solving the problem at hand.
• Develop an attitude to be ok with underdelivering on performance, but only when it fits customer expectations.

The problems introduced with story points:

• Clashes over virtual numbers. “8!”, “13? why 13?”, “allright, 5!”. It’s a crazy cacophony of hurling numbers around.
• Story points get abused by management as performance indicators on either team performance, inter-team performance, or personal performance.
• Teams learn to game story points to manipulate how they look to outsiders. This obscures actual performance.

In the end, there is a practical rule:

Software development takes as long as it needs to take. Story points do not change that reality, estimates and planning do not necessarily improve the lives of people in a project. It only works if you make your practice more predictable, and this means reducing innovation and evolution.

Focus on this instead:

• Do we deeply understand the customer problem context, or our technical challenges for this project?
• What is the minimum scope we can get away with? How can I trim more scope for the first milestone? Keep every milestone as short as possible.
• Get the result in front of your stakeholder as soon as possible. Shorten the customer feedback loop.
• Build your new project end-to-end first, connecting backend to frontend, then worry about details. This means, focus on your “hello world” first. From there, the majority of your infrastructure is in place, the hardest problems emerge emerge and you haven’t put yourself in a spot where you have to spend weeks recoding things, because it doesn’t fit your assumptions.
• Build your app “in parallel”. Start with the frontend API and provide mock data. Now backend and frontend can work in parallel and you have just established your “hello world”. If you build the API last, many bad things happen: frontend engineers are put under pressure, problems in API design turn out late, impact on backend data structures surface late. When you build in parallel, you’re probably also building end-to-end first. Empower others.
• Avoid too many new technologies at the same time. If you need to move data from one database to another that you poorly understand, and you decide to do this with a new programming language, and fix a lot of issues along the process, eliminate some uncertainties.

Long feedback loops

A feedback loop is a system, where the output of that system is used to tweak the input to that system.

If you are busy working on a program, and in order to run the program, you have to go through a five minute build process, you don’t get feedback about the correct functioning of the code until five minutes later.

Compiling

I give examples of specific feedback loops to fix in the Optimize section below.

Doing work that does not contribute to your end goal

Some engineers follow best practices, because this is the common understanding of how things should be done. Every bureaucratic action or thing you do during the day needs to have positive value somewhere in your organization. When you do not have external dependencies, ask yourself if the best practice you’re following is useful to your team and project.

Maintaining three separate environments is costly. It is often prescribed and required by architects to follow because of best practice. Engineers may feel ashamed if they bypass the process. But if you do not catch any bugs or issues as you move through those stages, you’re just following the steps to get your software promoted. It slows down delivery, it causes people to hold off on promoting to production, because the process is lengthy and takes a lot of time.

Either scrap unnecessary manual procedures or automate them.

Automate everything

Manual, routine procedures are boring and reduce work excitement. It introduces possibilities of failure and mistakes. And routine tasks aren’t a good fit for engineers, who should optimize their time for working on solving complex problems, not pushing buttons to promote software to production.

How to optimize

Trim all the feedback loops you are aware of

This is at the very core of agile.

Waterfall software development helped us to evolve the field of software development. Before waterfall, engineers and stakeholders didn’t have a great way to communicate exactly what is needed. Waterfall created a process for software development, where software development took a bit longer than now, to give engineers correct upfront information what was needed. The cost of making changes was much higher than it is now. Part of the need for documents is of course also rooted in legal needs.

But the waterfall process has an inherent long feedback loop, where the design takes a long time to review. Because of this long feedback loop, another process called “change management” was needed. In the end it is difficult not to end up with a long list of bugs, scope changes, change requests, budget overruns, complicated political discussions and lack of budget to fix it.

Agile (attempte to) address that by significantly shortening the feedback loop. This eliminated the need for a complicated change management procedure. We’re probably very close to a system where development is as good as it can be. But also in agile, there are low and high performing teams.

What sets the good performers apart from bad performers is their ability to optimize their feedback loops across the software development lifecycle.

Examples of feedback loops to fix

• The sprint retro: The existence of a sprint retro means that engineers in a team make different decisions towards “fixing things”. Problems in the build process that frustrate engineers pop up in the retro, then an action point is created. What happens after that really depends on the organization, the action item might be indeed picked up next sprint, in other organizations it depends on the mood of the Product Owner, who doesn’t feel the same pain as the engineer. Without the sprint retro, fixing things as part of a regular development lifecycle becomes part of work. The thing there to track is whether engineers understand the tradeoffs, customer expectations, project constraints.
• The MVP: When you start building software, the MVP is where the customer sees the product for the first time and is able to deliver feedback. We often use mock-ups and maybe intermediate versions.
• Building end to end first: This feedback loop is about you seeing your product working end to end before you work out all the backend details, or the frontend polishing. If you do all the backend work first, and then find out your design doesn’t fit one stage in the pipeline, you might end up recoding it all differently, redoing a lot of the code that you worked on the weeks before.
• Proof of concept: If you have high uncertainty about a specific component or technology that you need to use, do a proof of concept of 1–3 days to explore that and build confidence. This allows you to change direction earlier. Only use the concept for that purpose, do not continue to develop it because it is fun or to discover anything else. Then just start building the actual product.
• Unit tests: If your unit tests are trusted, they provide a great measure of confidence in software development. But the time in CI you should allow this to take must be short, otherwise it will eat into your development time. Consider both coverage and execution time.
• Local development: How long does it take to build and run your application locally? With Docker this can take a long time. Short local build times not only cost money for direct time loss, they also cost engineer concentration and focus, which eats into engineer productivity. Waiting a long time for a build to finish makes people distracted often, interrupting “the zone”. Cloud provisioning for example often underestimates the processing time in the cloud for provisioning, which can lead to 2–3x longer development time than estimated before we know that ‘stuff works’.
• Data diffs in CI: This is where we at Datafold operate. The feedback loop to really understand if SQL changes didn’t break the data can be as long as a day. The impact of data breakage is massive: not being able to use the data for decision making, rerunning the build might take an hour or longer, getting the data fixed takes time. You can shorten the feedback loop on data breakage by diffing the data in the pull request.

Choose a short sprint / cycle duration

Longer development cycles accumulate more needs for changes and change requests. Scrum teams commonly use a two-week cycle. I clearly remember the reasoning not to make it shorter:

We cannot fit all the Scrum ceremonies in a cycle of one week.

Remember what I said about: “Are recurring meetings useful?”

When you eliminate the sprint retro, you created enough room for weekly sprints. Then you have to figure out how to deal with what you normally deal with in retros. Here are some suggestions:

• Be very practical and fix things as you go. This does require a team that can make good trade-offs and balances between business impact and solving technical problems. Usually this means increasing communication about customer impact, expectations and constraints, thus more customer and project context in planning sessions and scoping documents upfront.
• Retro sessions are run by populating a board with successes and improvements. Why build that board from scratch every time? Maintain a board on the wall with frustrations about build process, execution time, etc. and find a way to “+1” it every time it pops up for you. If you make frustrations public as they happen, it contributes to immediate

The benefits of reducing your cycle from two weeks to one week became very clear to me:

• You have doubled the touch points between engineers and stakeholders. That is one other checkpoint where it’s easier to make roadmap amendments. In two-week sprints, if the plan needs changing, the sprint “is interrupted” and everyone is frustrated. Some engineers might have put off their work until the second half of the sprint and feel frustrated not being able to fix that. How do engineers schedule their work for the sprint in your team? Which issues come first? How does that affect things on sprint interruptions?
• Two week cycles leads to engineers procrastinating the first week. The second week they get started, discover unexpected complexities late and are under pressure to deliver. This reduces quality and increases technical debt.
• The cadence is more routine. A team member agenda is consistent every week and doesn’t have an alternating rhythm.
• You can only do this after implementing Continuous Delivery. If you do a lot of manual stuff to prepare the release, you need a bi-weekly release schedule.
• If you have uncertainties about a story, you don’t have a two week delay because of the process. The project owner probably doesn’t like suggestions for a proof of concept, because throughout the process, the outcome of the concept leads to new stories that are only picked up the next sprint. This leads to conflict and long planning sessions. If you do a poc one week, you wrap this up by friday and monday you have a new path for continuation of the project.

One week planning sessions reduce friction and conflict.

Choosing performance metrics

Pages and pages have been written on which metrics to track for engineers. In agile, this unfortunately has become the velocity metric.

In defense of using the metric, we say that you should never optimize for a single metric, but take a number of metrics into account and then make the assessment from the combination. (exact formulas for the sum do not exist, so that’s where the subjective evaluation begins).

Any metric that becomes loaded by performance evaluations can become a target for gaming the metric.

• If you create a metric where people can influence the volume of the metric itself, you create a system that obfuscates what the metric was initially intended to do. Do not use velocity metrics for performance evaluations.

A poorly designed metric can lead engineers to optimize towards the wrong goals.

• Years ago some companies used “SLOC” counts, the number of lines of code you write, as the performance evaluation metric. Your salary was impacted by that. The total lines of code significantly bloats and confuses engineers and quality goes down as a result, as well as maintainability. Less code is better.

Metrics designed for specific teams, but not the whole organization, leads to conflicts and “gating” between teams.

• If you have two teams, an SRE teams that gets punished for downtime and a development team that gets rewarded for the frequency of releases, you have created a situation where the SRE team requires forms and confirmations tests were carried out to guarantee the functioning of the application. The metric creates a situation of opposing interest in your organization with all kinds of discussions and politics around it.

DORA metrics

The book “Accelerate” of 2018 did a lot of research on high and low performing teams and they found some commonalities. Their suggestion for metrics to use are called the “DORA metrics”. DevOps Research and Assessment team. Before implementing these metrics, read this book first.

• Deployment frequency: How often a software team pushes changes to production
• Change lead time: The time it takes to get committed code to run in production
• Change failure rate: The share of incidents, rollbacks, and failures out of all deployments
• Time to restore service: The time it takes to restore service in production after an incident

There are plenty of articles explaining these metrics. It is also important to realize that you need to understand the context in which these numbers are used, the metrics by themselves do not lead to optimization, you need to understand how to interpret the metrics and which actions and things lead to improvement or worsening.

Principles

Let’s list the final principles here to not make this article too long.

• Trim feedback loops. The time between action and feedback to that action needs to be as short as possible.
• Use a short cycle for development. It allows for a higher rate of changes to occur and removes a lot of politics.
• Choose good performance metrics. The definition of your metrics functions as a steering wheel for your development organization.
• Rethink the use of story points. Understand why you do it, do not just follow the method, because the method says so. Someone invented them and thought it was a good idea. That was also just a person.
• Get rid of unnecessary recurring meetings.
• Get rid of work that does not contribute to any organizational goal. Don’t just follow “best practice” because it’s an industry standard. Apply it when it makes sense for your organization only.
• Automate routine tasks. Deployment, Continuous Integration, monitoring, observability, security checks, kubernetes operators, etc.