7 Principles of Software Testing
Did you know in 2016, the recorded software failures cost the worldwide economy of $1.1 trillion. This was more than the combined GDP of South Asia that year. The root cause is the lack of quality in software. Quality means — “a holistic, emergent property of a complex system in a particular relationship to an observer. ’’ or simply a scale of goodness and/or value for a subset(s). This means quality depends on each person and context. And the rule of thumb is quality comes with a cost, not to be misunderstood that quality costs high but actually means the cost of not creating quality is high.
Quality is free. It’s not a gift but it is free. What costs money are the unquality things- all the actions that involve not doing jobs right the first time.
-Philip B. Crosby
Software Quality
As stated earlier that quality depends on the eye of the beholder, let’s analyze different familiar scenarios but often conflicting ideas about Software quality:
- “Zero defect is high quality” — to a manager who would be criticized for those defects.
- “Lots of features is high quality” — to marketers who believe that features sell products.
- “Elegant coding is high quality” — to developers who prioritizes opinions of their peers.
- “High performance is high quality” — to a manager who wants to compete with similar products in the market.
- “Low development cost is high quality” — to project managers who are on a tight budget.
- “Rapid development is high quality” — to users who are eagerly waiting for new updates.
- “User friendliness is high quality” — to a user who frequently uses a software
If one has to define Software Quality in general, it is the combination of factors that represent the behavior of a system. Correctness, Reliability, Efficiency, Testability, Maintainability, and Reusability are some of the quality factors.
Testing
When a product doesn’t behave as intended, then definitely it’s a low-quality product. So the process that determines if the product works as intended is called Testing. Software testing is a process, to evaluate the functionality of a software application with an intent to find whether the developed software met the specified requirements or not and to identify the defects to ensure that the product is defect free in order to produce the quality product. A part of the software that causes the system to produce an incorrect or unexpected result is called Bug. The motive of Software testing is to detect these bugs.
The 7’s
These are just the guidelines for achieving quality software.
1. Testing shows the presence of defects
This principle describes the definition of testing. If a culprit looks innocent doesn’t mean that he really is. This means that if your software passes all test cases that doesn’t mean your software is bug-free. Testing reduces the probability of undiscovered defects remaining in the software but even if no defects are found, it is not a proof of correctness.
Takeaway: Testing shows the presence of defects and doesn’t specify its absence.
2. Exhaustive testing is impossible
Yes! Exhaustive testing is not possible. It is simply not possible to test all possible cases and combinations, of course, if this is not a trivial case. Instead, we need the optimal amount of testing based on the risk assessment of the application. So to test effectively in such condition we can first determine boundary values and partition data to valid and invalids. Then pick a few valid and invalid for testing and assume output will be similar for data on the same group
Suppose you have an input field which accepts productive age (16–65). So to test this first we need to define boundary values in this case, less than 16 and more than 65 are invalid, and between them are valid values. So here we did a special process called Boundary Value Analysis. Next, we need to partition this data to valid and invalid sets called Equivalence Partitioning.
Takeaway: Do effective testing, not exhausting testing.
3. Early testing
It is important to start testing as early as possible and anticipate possible errors that the developer can make. Why?
- Cost of Bug fixing
Early you find a bug, less will be the cost of fixing that bug. Here cost includes effort, time, money, and other resources. The above diagrams display defect fix cost along SDLC.
- Maximum bugs originate in Requirement
According to research, 56% of defects are originated in the requirement, 23% comes from the design phase, 7% comes in the development phase, 14% defects originate in regression testing and other sources.
- Better understanding of requirements
The earlier you start testing, earlier are your ambiguity resolved. A well-understood requirement means you deduct 56% likelihood of creating bugs.
Takeaway: Early to test is early to quality raise.
4. Defect clustering
Bugs are not often distributed evenly throughout an application. Stating with the Pareto principle, 80% of your bugs can be found at 20% of components.
This basically happens because an area of the code is complex and tricky. Test designers often use this information when making the risk assumptions for planning the tests, and will focus on these known areas that may also be called Hotspots.
In order to effectively test the product, we should distribute our efforts in testing according to the real density of bugs in the product modules(that 20 %); if we test for the first time, it is proportional to the expected density. Over time, the trend with the clustering of bugs may change from module to module. This should be monitored and redistributed in further testing.
Takeaway: Invest 80% of your effort in finding those 20% culprits.
5. Pesticide Paradox
Pesticide Paradox refers to the repetitive tests that testers run in projects so that it stops being effective in catching bugs. This kind of test may cause the issues of not discovering new bugs outside that modules. For example assume a module tester executes some test case and raised many bugs, so a developer will be extra careful in those modules. Hence re-executing the same test case again and again for the module will no longer help to find more defects.
So how not to be affected by this problem:
- Keep track of product changes and their indirect effects in your application.
- Discontinue tests that are not effective.
- Modify your test data(frequently).
Takeaway: Similar testing increases Bug immune.
6. Testing is context dependent
The testing methods depends on the context in which it is to be used. The testing for an e-commerce platform will be different from the testing of an off-the-shelf commercial application(COTS). All software, even for the same application, are not identical. They can be different in techniques, and it will affect the testing methods that need to be deployed for a specific purpose. For example, testing software for an online platform will be different from the testing software for a physical store.
Takeaway: No two applications are the same, so should be their testing.
7. Absence of error — fallacy
How likely is a user willing to pay for your bugless product which he doesn't need? It’s possible that software which is 99% bug-free is still unusable. This can be the case if the system is tested thoroughly for the wrong requirement. Software testing is not merely finding defects, but also to check that software addresses the business needs. The absence of Error is a Fallacy i.e. Finding and fixing defects does not help if the system built is unusable and does not fulfill the user’s needs & requirements. So try focusing on fulfilling user requirements rather than creating never used bugless feature.
Takeaway: Fulfilling requirement counts more than Bug.
