The Outrageous Cost of Skipping TDD & Code Reviews

Eric Elliott
Dec 14, 2016 · 6 min read
Image for post
Image for post
The Color of Money — Chris Potter (CC BY 2.0)

Update Jan 26, 2019: It has been brought to my attention that the claim “fixing a production bug may cost 100x more than fixing a bug at design time” is apocryphal. I still stand by the positive ROI and timesaving powers of TDD, but you should read this as a subjective opinion rather than scientific fact. Our industry is in great need of better data collection on the impacts of TDD and other quality control measures. This would be a great subject for research collaborations between universities and large software organizations.

In recent years, I’ve had more and more companies ask me to speak on the benefits of Test Driven Development (TDD) and share advice about how to implement a more productive quality process.

TDD is the process of writing automated tests to ensure that code works before writing the implementation. You write a test, watch it fail (red), write the implementation, watch the test pass (green), and refactor if needed. Repeat the cycle as you build out the system.

The process has been studied in depth, and has proven itself to be very useful to increase the quality of software. But did you know that it also saves organizations a lot of time and money?

One of the primary reasons managers cite for waiting so long to implement TDD is the cost. It’s common for initial project build-outs to take up to 30% longer with TDD.

What those managers need to know is that TDD reduces production bug density 40% — 80%, and that makes all the difference. More bugs in production leads to a dramatic rise in maintenance costs.

Due to exponentially escalating costs of rework, meta-work (work about work), responsibility hand-offs, interruptions, and eventually customer support, bug triage, and maintenance, fixing a production bug may cost 100x more than fixing a bug at design time, and over 15x more than fixing a bug at implementation time.

Note: This claim has been challenged, but there’s no question that software quality improves developer productivity. See “The Economics of Software Quality” for an in-depth understanding of the available data on software quality, and the impact software quality can have on your organization. We need a lot more studies on software quality measures so we can make better informed decisions about how to run development teams.

Image for post
Image for post
Relative costs of fixing bugs

Using these figures, let’s look at the relative costs of a fictional project with and without TDD. The project will take 1,000 hours of up-front implementation without TDD, not counting maintenance:

Image for post
Image for post

In our fictional example, we saved 623 man hours, or, for a team of 4, about a month of development. If we paid each developer the US national average ($95k), and factored an average of 30% on top of salaries to cover benefits, we reach a savings of almost $37,000 USD.

Of course, this is a fictional example making a lot of assumptions. Your mileage will vary based on lots of factors, including your team’s experience with TDD, the presence or absence of other quality measures (such as code review), etc…

Here are the assumptions made:

  • Production bugs without TDD: 100
  • Production bugs with TDD: 40 (60% fewer, middle ground between 40% — 80%)
  • Average time to fix bug at implementation (TDD) phase: 1 hour (this number is only used to derive the production bug fix cost)
  • Average time to fix a production bug: ~15 hours

Playing with any of those variables will obviously alter the results. What is pretty certain is that TDD saves time & money as maintenance costs get factored in — a lot of time & money.

The relative results of this fictional example ring true to me based on my experience with real production projects, both with and without TDD.

Code Reviews Have Similar Effects

That’s pretty extraordinary, but the bug catching benefits of code reviews just don’t jive with my personal experience. I find automated tests far more useful at catching bugs — so how do we explain the extraordinary effectiveness of code reviews?

Knowledge sharing. When you employ code reviews, your team self-corrects when developers are using anti-patterns and other poor practices. At the same time, they’re sharing effective patterns, teaching each other better ways to do things, and teaching each other how to write more readable code.

The net result of that is that your junior developers quickly rise to the level of the smartest developers on the team, and your whole team velocity improves. It’s hard to over-state the value of a mentorship culture on your team.

The Cost of Interruptions

Fixing a bug in production is much more expensive than fixing a bug during development. To understand the magnitude of the difference, you must first understand the cost of interrupting a developer.

A production bug fix frequently means an interruption in the context and cadence of feature development. In other words, developers are pulled out of the context in which they’re currently working and dumped into the context of the bug, where it takes time to absorb the related code and flow, diagnose a root cause, fix the bug, and then reabsorb the context of what they were working on before.

Each context switch can cost up to 20 minutes of developer productivity, but the bleeding doesn’t stop there. Interrupting a developer to fix a bug is likely to cause more bugs in the code they were working on before they were interrupted. According to Microsoft Research, an interrupted task takes about twice as long to complete, and contains twice as many errors as an uninterrupted task.

In other words, the cost of fixing bugs that get released in production isn’t just about the cost of fixing the production bug. Interruptions increase the cost of current development work, and introduce more bugs that will eventually need fixing, too.

On a related note, multitasking is a really bad idea. To be productive, developers need to focus on one thing at a time.

Conclusion

Next Steps

Level Up Your Skills with Live 1:1 Mentorship

  • Live lessons
  • Flexible hours
  • 1:1 mentorship
  • Build real production apps
Image for post
Image for post
https://devanywhere.io/

Eric Elliott is the author of “Programming JavaScript Applications” (O’Reilly), and cofounder of DevAnywhere.io. He has contributed to software experiences for Adobe Systems, Zumba Fitness, The Wall Street Journal, ESPN, BBC, and top recording artists including Usher, Frank Ocean, Metallica, and many more.

He works anywhere he wants with the most beautiful woman in the world.

JavaScript Scene

JavaScript, software leadership, software development, and…

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch

Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore

Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store