Thoughts on Efficient Test Design #1
Testing is a significant challenge in the development process. Especially with business-critical systems, the testing effort can take up 40 to 60 percent of the costs. Although there are more and more technologies to automate test execution, testing remains mainly human work, as testers hand-write test cases based on requirement specifications. For many, this has become a creative job, and these people specialize in software test engineering — in a way that this activity may sometimes be more complex than programming. Many customers even see testing as a parallel software project and experience this as a duplication of work.
The classic approach to test design, where we write test cases by hand, poses many challenges. Some are listed below:
- High effort: Classic test design is labor-intensive, expensive, and has a long lead time. The number of test cases required for the desired test coverage must be more extensive for complex systems to write by hand.
- Incomplete test coverage: Classic test design only sometimes thoroughly tests and does not give a good indication of test coverage, making it challenging to have confidence in the quality and correctness of the delivered systems.
- Difficult to maintain: Classic test design requires much extra work when the specifications change. Testers must check and update test cases manually. As a result, the quality of the tests often needs to improve over time.
Efficient Test Design
Many books, articles, and blog posts claim that you either spend more time and money in testing to achieve higher quality or save money, but your software quality still needs to improve. This is not true, and there are two main cost elements in the software development lifecycle (SDLC) that teams can modify: test and debug, and neither is linear.
The cost of testing increases when a team wants more confidence in software quality — with more test coverage or detected bugs. In the beginning, simply increasing the number of tests will provide a linear effect — meaning doubling the designed/executed test cases will likely double the coverage and found bugs. However, at some point, we’ll need to test the elements of the input domain leveraging combinatorial methods, e.g., testing some partitions, their boundaries, then pairs or triples of boundary values. Even with all these extra tests, we’ll be unable to find a lot of other bugs or increase coverage too much, so there’s little reason to design or even run the tests.
The other factor is bug fixing and the costs involved. Fixing a bug late in the software lifecycle is much more expensive, and the correction cost is exponential in time, showing polynomial growth. If we find defects early enough in the lifecycle, we can drastically reduce the total correcting costs.
The question is — does the optimum cost assure acceptable code quality at the feature/software release? Assuming we applied an ineffective test design, we will focus on test execution to ensure defect discovery. Even with the increased focus on the test execution phase, generating increased costs, the number of undetected bugs would remain high due to uncovered/obscure requirements. Even if you reach optimum, quality may not be acceptable.
On the other hand, if very efficient test design techniques are applied, the optimum will be at a point where the detected bug ratio is much higher, resulting in acceptable quality.
This means that although different features require different levels of test design, there are better strategies to reduce testing efforts and costs than you should consider these two factors together. An efficient test design technique is also necessary for the SDLC cost to decrease and the code quality to become acceptable simultaneously.
Roughly speaking, more complex code and higher risk requirements raise the bug-fixing costs if the testing effort (and, as a consequence, testing cost) remains unchanged. Since this additional bug fixing occurs later in the lifecycle, the code quality at release will be poorer. Therefore, a riskier or more complex feature must be tested more thoroughly, i.e., with higher testing costs.
