Observations of High-Performing Engineering Teams

Pair programming… a great example of a high-leverage activity.

I’ve had the opportunity to work for software organizations big and small. I started my career at a Fortune 100 company that boasted over 200 software development teams. Later, I joined a tech-startup as employee #14. While the companies were radically different, there were definitely commonalities between the highest-performing software teams.

High-performing engineering teams are the right blend of talent, culture, and practices. Like any good recipe, too much of one thing and not enough of the other leads to a less than optimal result. Very skilled practitioners who have lacked discipline in a certain area can be less effective than other teams who have healthier and a focus on continuous improvement.

I’ve condensed a few of my observations here.

They focus on high-leverage activities

High performing software engineering teams always work on high-leverage activities. High-leverage activities are those that provide a longer-term, recurring benefit; above and beyond the needs of today’s requirements. Simply put: high-leverage activities are ones that make the team better.

Some examples of high-leverage activities are:

1. Peer-reviewing source code
   Code reviews are generally the primary vehicle for knowledge dissemination on a team. Additionally, this will lead to the internal structure of the code being improved, defects being caught, best-practices being enforced, etc. The benefits of code reviews have been studied and written about, yet many teams don’t adopt it.
2. Interviewing candidates to add to the team
   Software engineering is a team sport. Anybody being added to a team should have some baseline technical knowledge, the ability to learn new concepts or methods and also be a cultural fit within the team. The implications of a bad hire are too great; therefore it’s important to consider multiple viewpoints on any hiring decisions. Specifically, the viewpoint of the people who will work the closest with a candidate should have a commensurate say in the hiring process.
3. Automating tasks or building tooling
   Many times there are feelings of guilt associated with building automation & tooling for the benefit of the engineering team. The thought of spending time sharpening the ax is seen as a fraudulent misappropriation of time. In practice, automation allows the team to work faster and scale beyond its current limitations.

After the initial investment, each of the above will have a lasting benefit to the team. For example, pairing an expert with a novice allows knowledge to transfer between them. In time, the novice will be able to take on part of the responsibilities that only the expert can work on today. The novice may be able to provide a new perspective on old problems. Additionally, the team has minimized a dependency on a particular person.

“Give me 6 hours to chop down a tree, and I will spend the first 4 sharpening the axe.” — Abraham Lincoln

For contrast, here are some examples of low-leverage activities:

1. Manually testing a set of scenarios every release.
2. Myopically fixing defect after defect without addressing the root cause.
3. Allowing a single person to be the sole-maintainer of an important part of the system.
4. Manually copying or deploying artifacts.

They are always testing.

“Good code doesn’t make a good system. Good testing does.”

Manual testing should be thought of as a low-leverage activity. Because of the intricate nature of software, any manual testing performed yesterday doesn’t provide any confidence about the state of the system as it is today. Manual testing has no recurring benefit and is simply accepted as a sunk cost.

Anecdotally, the lowest-performing engineering teams always seem to have issues with software quality. Why is that?

Let me introduce to you the frustrating death spiral that is most enterprise development teams:

1. Because the quality is bad, they spend more time manually testing.
2. Because the manual testing cycle takes a long time, unvalidated work-in-progress piles up.
3. Because unvalidated work piles up, testing uncovers more bugs. These bugs take longer to fix since the developers have moved on to new work-in-progress.
4. Because bugs are now found late in the cycle, it’s a mad scramble to fix them all, leaving no time remaining to perform clean up activities.

And this foolish cycle is repeated ad nauseam like a dog chasing its tail.

To break the cycle, testing should be performed as soon as the code is in a workable state. The key is to detect issues quickly, and in doing so preventing the build-up of unvalidated work which will have an even greater downstream impact. Even if only a subset of validation can be performed, you’ll have at least some amount of confidence that things are working as designed.

The benefits of test automation are pretty much globally accepted at this point. When pressured to start test automation, many teams who haven’t seen the light yet have an unusually common retort in hand:

“I don’t have time to automate my tests!”

Citing bugs or other urgent issues that require their full attention, many times opportunities to write test automation are not taken advantage of. Ironically, the reason they don’t have time to write automated tests is because they haven’t automated their tests.

The size and complexity of software systems tend to grow over time. Using math, you can assume that as size and complexity grow, so should the number of test cases. Without automation, as the number of test cases grow, so does the length of time required to completely test a system. And we’ve now introduced the spiral of doom that I introduced at the start of this section.

“You don’t have time to write automated tests because you don’t automate your tests.”

A quick side note: a software component’s testability is usually a good proxy for its internal quality. So if you’re struggling to write a unit test for a component, that’s usually a pretty good code smell that the design of that component is lacking in some way.

They learn from others and stay current with technology

Many times the difference between an intractable problem and a simple one is a heads-up use of a framework or new technology. The software development world is very large and it is highly unlikely that you’re facing a problem that someone else hasn’t solved already.

Keeping up with new technology is easy to do. For example, common ways of doing so are:

1. Following & sharing Twitter, Reddit, YouTube or RSS feeds.
2. Having regular lunch-and-learn sessions to discuss interesting or relevant technologies.
3. Sharing lessons learned from blameless postmortems.
4. Having an in-office book club.
5. Encouraging team members to attend or speak at user groups or conferences.

Especially in task-driven organizations, the team may not feel like they have permission to entertain such things. If you’re in a leadership role, you may need to do some special nudging to get people thinking this way.

They communicate well.

High-performing teams are great communicators, and as such, they are always operating off the same sheet of music. They have a shared vision of success and collaborate often to ensure that they are moving in the right direction. Usually they’ll have a daily standup and use a collaboration tool like Slack to stay in-sync throughout the day. It’s easy to communicate when things are going well.

Unfortunately, unexpected twists and turns can happen. Priorities may change mid-flight or deadlines may be in jeopardy. Team members may make mistakes (gasp!) causing re-work or a change in plans. Worse yet, some members of the team may be inhibiting the progress of the larger group. Tough conversations may need to be had.

High-performing teams face these challenges head-on and are very transparent about them. They don’t to hide information or (even worse) radiate false information to save appearances. These small acts of misinformation, while well-intended, accrue over time and do more damage than good. Difficult issues can be great learning experiences for the team. Taking away the opportunity to work through issues robs a valuable learning opportunity.

1. Status should be big and visible, data-driven and not massaged to paint a pretty picture. Teams should leverage dashboards to maintain a common operational picture.
2. Leaders of high-performing teams are upfront about expectations and values, and give regular feedback — both positive and negative.
3. The team decides on and enforces it’s own best practices and standards (presumably using peer reviews). These standards reflect knowledge gained from previous lessons learned.
4. Blocking-items are clearly called out and swarmed on to keep forward momentum.
5. Risks are discussed openly, with the intent of mitigating them before they turn into issues.
6. Post-mortems or after action reports should be openly available so others can avoid repeating the same mistakes.

If you enjoyed this article, please consider sharing it on your social networks!