What is the ROI of my test automation?
(TL;DR - calculate your ROI here!)
In the era of Agile software development, test automation has gone from a ‘nice to have’ to a critical component of delivering quality software at velocity.
Business decision makers generally agree that test automation is a good idea, but may hesitate to commit appropriate resources. Uncertainty around the costs of building and maintaining a test automation framework, especially when budgets and deadlines are tight, can lead to questions about how much value it delivers.
For many stakeholders, a qualitative explanation of how an increasing number of manual regression tests each sprint eventually puts a stranglehold on the team’s ability to deliver is enough to justify investment in test automation. For others, especially those outside of the technology organization, a quantitative justification may be helpful to get buy-in.
In this article, I will demonstrate a way for you and your teams to communicate the value of test automation to fiscally-minded decision makers.
Stakeholders will understand that while there are upfront and ongoing costs, the value gained from replacing manual regression testing over time will make the investment worthwhile in the long run.
The Value of Test Automation
Much of the value of test automation stems from running automated tests multiple times¹.
If tests are only run a few times, it may be faster and more cost-effective to run all the tests manually, rather than paying upfront costs of either purchasing a test automation tool or building a framework from scratch and coding tests.
However, as automated tests are run repeatedly against new builds a break-even point is reached. From this point on, all future runs represent ever-increasing positive ROI and often increased test capacity and coverage (that is, the ability to perform testing they previously may not have had time to perform manually).
In the example below, break-even is reached after 25 test automation runs, and by the 50th run it has reached an ROI of about 1.75, meaning that test automation had delivered 75% more value than what had been invested into it.
Calculating ROI
(If math isn’t your thing, please feel free to skip to the Test Automation ROI Scenarios section)
Here, ROI is calculated as savings gained by replacing manual regression tests with automated tests divided by the cost of investment into test automation:
ROI = Savings ÷ Investment
Since ROI is a unitless number, it really doesn’t matter whether the savings and investment amounts are in dollars or time. For ease in calculation, minutes will be used because most of our inputs are in the form of time.
One way to include monetary values in our estimates would be to convert dollars to minutes by dividing by the hourly bill rate² of the person doing the test automation work and then converting hours to minutes.
In a similar way, test automation cost savings can be estimated by subtracting investment costs from savings (both in minutes), converting the resultant number of minutes into hours and then, multiplying by the hourly bill rate³.
Cost Savings
Savings is the difference between the cost of running a set of tests manually versus running the same tests automatically a number of times over some period.
Savings = (⁴time to run one manual test case - time to run one automated test case) * # of tests * # of runs
- The number of tests is an average over the time period. If the test suite started at zero tests and grew linearly to 200 by the end, then the average would be 100 tests.
- The faster automated tests run, the sooner they can reach positive ROI. This makes a case for using integration tests³ instead of End to End (e2e) UI tests where possible since these often run 5 or 10 times faster, and tend to be more reliable.
Test Automation Investment
Investment is the summation of fixed and ongoing costs of test automation including time spent building and configuring the test automation tooling or framework, and time spent coding or maintaining automated tests.
Investment = time to build framework + (⁴time to code one automated test * # of tests) + maintenance costs
- If there are purchase or licensing costs, they can be added here after converting dollars to minutes by dividing by bill rate as discussed above.
Maintenance costs are the costs of investigating and fixing failed tests (both false positives and actual failures). Flaky tests and development bugs alike contribute to the maintenance cost, as investigation is required to determine if the feature or the test is at fault. The more frequently tests fail, the longer it takes to break even on ROI.
Maintenance costs = ⁴maintenance time for one failing test case* % of failing test cases per run * # of test cases* # of runs
- We assume an average percentage of failures across all tests
Test Automation ROI scenarios
The example above was based on the following scenario:
In this case, break-even was reached after 25 runs, and after 50 runs the ROI was about 1.75.
Let’s take a look at other scenarios and see how ROI and time to break even are affected:
Scenario #1: Test automation is only run 10 times
After 10 runs perhaps the project ended, or the test automation framework was abandoned. In either event, only about 45% (ROI = .45) of the investment has been recouped and it would have been more cost-effective to go with manual regression testing. The assumption here was an average of 150 tests, but if the suite were still ramping up the number of tests, the ROI would have been even lower.
Scenario #2: Building the test framework takes twice as long
We have our original scenario again, but this time instead of 2 weeks it took 4 weeks to get the test framework ready to start coding tests. At first glance, it may seem expensive to pay someone for an extra two weeks of work, but after running the test automation framework 9 more times (34 instead of 25) the deficit is made up and ROI breaks even. After 50 runs the ROI is about 1.4.
Scenario #3: Tests take twice as long to run
In this scenario, each automated test takes a minute to run, instead of 30 seconds. Despite longer execution times, it only takes 4 additional runs (29 instead of 25) before ROI breaks even, and after 50 runs the ROI is just under 1.6.
Scenario #4: Tests fail twice as often
Another main factor in test automation ROI is test robustness. Any time a test fails, whether actual failure or false positive, time is required to locate, diagnose and remediate the issue. If a large number of tests often intermittently fail (due to timing issues etc.) it can have a significant impact on ROI. In this model it takes 6 additional runs to break even (31 instead of 25) with an ROI of 1.35 after 50 runs.
Since one limitation of the model is that it does not factor in time spent rerunning test automation each time there is a failure to confirm that the failures are real, the actual ROI is likely to be somewhat lower than above.
Scenario #5: Framework and tests take twice as long and fail twice as often
In this case, we have somewhat of a worst-case scenario where each of the issues from scenarios 2- 4 impact the project. Our first inclination may be to add up the delays from each scenario to estimate our break-even point. Adding these together we would get 19 additional runs and it follows that it might take 44 runs to break even.
Unfortunately, the confluence of factors deals a greater hit to ROI than the sum of each and we just barely scratch out a positive ROI on our 50th run.
Last Scenario: All tests replaced with service level / integration tests
Depending on project type, it may be possible to replace some manual tests with automated service or integration tests. In this case, these tests would no longer be run through the UI, often yielding faster run times, more robust tests and less maintenance time. In this scenario, our break-even point moves left to 19 runs, and the ROI after 50 runs is a whopping 2.4!
For your application it may not be possible to replace all manual tests with service or integration tests, but we consider it good practice in accordance with the Test Automation Pyramid to avoid relying fully on E2E UI tests, and to emphasize service and integration tests as much as possible.
Limitations of this model
Our ROI model is intentionally simple to keep the number of inputs low and make as generally applicable as we can. As a result, there are other factors that can impact ROI, but are not incorporated in the calculation.
These include:
- Unit tests: Unit tests are a valuable development activity that enhance the code quality. By nature, they are too narrowly defined to replace manual tests, but their value is implicit in the model as a potentially lower number of failures to investigate. They are not included explicitly in the ROI calculation.
- Test coverage gains: Having the ability to run automated tests fast or during off hours can result in greater test coverage or capacity. Tests can be run that there would not be enough time to run manually, or additional manual tests can be run while automated tests are running. Gains such as these are not factored into the ROI metric.
- Test rerun costs: When flaky tests have to be repeatedly run against the same build, these re-runs do not count towards ROI.
- Other costs: Additional infrastructure and operational costs not factored in can be converted to minutes and added to the Investment total.
- Converting ROI to dollars: It is a simple calculation to convert ROI to dollars as shown above. This can be useful for communicating with business decision makers, but its accuracy may be limited, since the model is very simple.
For greater accuracy, this model can be customized to factor in these or other costs relevant to your organization.
Conclusion
Test automation requires up-front and ongoing investment that pays itself off with use, and significant gains can be made by implementing service level or integration testing as part of a holistic test automation strategy.
Relying on manual regression testing as a product grows, not only fails to realize these gains, but puts the product and possibly the organization at risk. Costs of testing and bug remediation grow with each iteration, and eventually product costs, quality, and time to market are impacted.
Using this ROI model, business decision makers can quickly and easily understand the value that test automation delivers over time, while technical team members gain insight into the effects of test speed, robustness, and maintainability on the value of their test framework. Both can use it to drive right decision making, leading to high value test automation in your organization.
Notes:
[1] Running test automation against the same build repeatedly does not contribute to value since there are no new regression bugs to be found between runs. If the tests have to be run repeatedly due to test flakiness, this contributes to cost of investment and reduces overall ROI.
[2] If multiple people having different bill rates are doing the test automation work, a weighted average of the bill rates can be used in proportion to their effort.
[3] Any kind of automated test that effectively replaces a manual regression test can be estimated using this method.
[4] An average time is used here. One simple way to obtain average test times is to use a triangular 3-point estimate, where a simple, typical and complex test is identified and times are estimated for each, added together and divided by 3.
