Update Jan 26, 2019: It has been brought to my attention that the claim “fixing a production bug may cost 100x more than fixing a bug at design time” is apocryphal. I still stand by the positive ROI and timesaving powers of TDD, but you should read this as a subjective opinion rather than scientific fact. Our industry is in great need of better data collection on the impacts of TDD and other quality control measures. This would be a great subject for research collaborations between universities and large software organizations.
In recent years, I’ve had more and more companies ask me to speak on the benefits of Test Driven Development (TDD) and share advice about how to implement a more productive quality process.
TDD is the process of writing automated tests to ensure that code works before writing the implementation. You write a test, watch it fail (red), write the implementation, watch the test pass (green), and refactor if needed. Repeat the cycle as you build out the system.
The process has been studied in depth, and has proven itself to be very useful to increase the quality of software. But did you know that it also saves organizations a lot of time and money?
One of the primary reasons managers cite for waiting so long to implement TDD is the cost. It’s common for initial project build-outs to take up to 30% longer with TDD.
What those managers need to know is that TDD reduces production bug density 40% — 80%, and that makes all the difference. More bugs in production leads to a dramatic rise in maintenance costs.
Due to exponentially escalating costs of rework, meta-work (work about work), responsibility hand-offs, interruptions, and eventually customer support, bug triage, and maintenance, fixing a production bug may cost 100x more than fixing a bug at design time, and over 15x more than fixing a bug at implementation time.
Note: This claim has been challenged, but there’s no question that software quality improves developer productivity. See “The Economics of Software Quality” for an in-depth understanding of the available data on software quality, and the impact software quality can have on your organization. We need a lot more studies on software quality measures so we can make better informed decisions about how to run development teams.
Using these figures, let’s look at the relative costs of a fictional project with and without TDD. The project will take 1,000 hours of up-front implementation without TDD, not counting maintenance:
In our fictional example, we saved 623 man hours, or, for a team of 4, about a month of development. If we paid each developer the US national average ($95k), and factored an average of 30% on top of salaries to cover benefits, we reach a savings of almost $37,000 USD.
Of course, this is a fictional example making a lot of assumptions. Your mileage will vary based on lots of factors, including your team’s experience with TDD, the presence or absence of other quality measures (such as code review), etc…
Here are the assumptions made:
- Production bugs without TDD: 100
- Production bugs with TDD: 40 (60% fewer, middle ground between 40% — 80%)
- Average time to fix bug at implementation (TDD) phase: 1 hour (this number is only used to derive the production bug fix cost)
- Average time to fix a production bug: ~15 hours
Playing with any of those variables will obviously alter the results. What is pretty certain is that TDD saves time & money as maintenance costs get factored in — a lot of time & money.
The relative results of this fictional example ring true to me based on my experience with real production projects, both with and without TDD.
Code Reviews Have Similar Effects
Code reviews have similar effects. In fact, some studies have found code reviews are more effective than TDD. According to a 1988 study, each hour spent in code review saves 33 hours in maintenance.
That’s pretty extraordinary, but the bug catching benefits of code reviews just don’t jive with my personal experience. I find automated tests far more useful at catching bugs — so how do we explain the extraordinary effectiveness of code reviews?
Knowledge sharing. When you employ code reviews, your team self-corrects when developers are using anti-patterns and other poor practices. At the same time, they’re sharing effective patterns, teaching each other better ways to do things, and teaching each other how to write more readable code.
The net result of that is that your junior developers quickly rise to the level of the smartest developers on the team, and your whole team velocity improves. It’s hard to over-state the value of a mentorship culture on your team.
The Cost of Interruptions
Why does it cost so much more to fix production bugs? Because once a bug reaches production, the cost is much more than the simple cost of fixing the bug.
Fixing a bug in production is much more expensive than fixing a bug during development. To understand the magnitude of the difference, you must first understand the cost of interrupting a developer.
A production bug fix frequently means an interruption in the context and cadence of feature development. In other words, developers are pulled out of the context in which they’re currently working and dumped into the context of the bug, where it takes time to absorb the related code and flow, diagnose a root cause, fix the bug, and then reabsorb the context of what they were working on before.
Each context switch can cost up to 20 minutes of developer productivity, but the bleeding doesn’t stop there. Interrupting a developer to fix a bug is likely to cause more bugs in the code they were working on before they were interrupted. According to Microsoft Research, an interrupted task takes about twice as long to complete, and contains twice as many errors as an uninterrupted task.
In other words, the cost of fixing bugs that get released in production isn’t just about the cost of fixing the production bug. Interruptions increase the cost of current development work, and introduce more bugs that will eventually need fixing, too.
On a related note, multitasking is a really bad idea. To be productive, developers need to focus on one thing at a time.
The next time somebody tells you they don’t have the time or budget for TDD or code reviews, tell them with confidence, “in that case, you really don’t have the time or budget to skip TDD”.
Get a better handle on TDD and its role in the software development process:
- “5 Common Misconceptions About TDD and Unit Tests”
- “Unit vs Functional vs Integration Tests”
- “5 Questions Every Unit Test Must Answer”
- “TDD in ES6 & React Webcast”
- Level up your team with mentorship: How to write more testable code, TDD, CI/CD process, and more
Level Up Your Skills with Live 1:1 Mentorship
- Live lessons
- Flexible hours
- 1:1 mentorship
- Build real production apps
He works anywhere he wants with the most beautiful woman in the world.