The 7 Principles of Software Testing
In the age of digital explosion, software quality is what sets apart one product from its competition. Read on for a few insights into the fundamental guidelines for software testing
We all know the importance of quality in any software system. The current pandemic-stricken world has particularly brought a lot of focus on how digitization is the way forward, with people across the world having to work remotely. Moreover, with the wide variety of digital platforms available — mobile, desktop, tablet etc. quality plays a very important role in simplifying and enhancing the entire customer experience ecosystem.
In a fast moving digital world where time to market is crucial, the quality of an end-product can prove to be a make or break deal for businesses. Of course, businesses cannot wait for too long for the final product to roll out since it is a question of the perfect timing or you lose the momentum and competitive edge. But quality cannot be compromised in the urgency to launch something in the market otherwise the results might be disastrous to fix (both cost-wise and reputation-wise).
There is a cost attached to quality and more specifically a cost attached to inferior/poor quality. This cost will only rise incrementally depending on how bad the quality is. Considering its importance, it is the responsibility of software engineering teams to ensure the quality of the product. One branch of Software Engineering — Software Testing, is entrusted with the responsibility of ensuring that the end-product being delivered is exactly what was expected. They have to validate that the system delivers what the user is essentially expecting from it (what we call as ‘requirement’ in software engineering parlance).
Software Testing is a disciplined profession which involves thorough analysis of requirements and expectations, analysis of past projects or activities, and the lessons learned about the quality of the product, as well as defining and following a robust process to find defects/bugs to ensure the best possible software quality. In addition, software testing also largely works on common sense and involves an imagination to conduct.
I won’t go into the specifics of defining ‘quality’, since it is quite self-explanatory. However, organizations such as ISO have devised certain internationally-agreed standards on how to define and measure quality. There are 7 principles of software testing that have been commonly accepted over the years all around the world, and serve as guidelines for testing. These principles have been accepted and adopted by the International Software Testing Qualifications Board (ISTQB).
These principles are listed under as:
1. Testing shows the presence of defects
2. Exhaustive testing is impossible
3. Early testing
4. Defect clustering
5. Pesticide paradox
6. Testing is context dependent
7. Absence of errors fallacy
Let’s go through each of them to understand what they mean.
Principle 1 — Testing shows the presence of defects:
The first principle basically talks of what testing is expected to do — find bugs.
Testing helps in uncovering bugs but it cannot be said that testing proves that there are no bugs in the system. A good amount of testing helps in finding bugs and reducing the probability of hidden defects/bugs in a system. This doesn’t necessarily prove the correctness of the system if there are no bugs found in it.
Principle 2 — Exhaustive testing is impossible:
The second principle talks about the possibility and more importantly the feasibility of exhaustive testing. Yes, you read that right! It is impossible to exhaustively test a system using all permutations and combinations of data and conditions. A proper risk analysis and determination of business priorities and critical use cases will help in optimal coverage of testing with satisfactory results.
Suppose I give you a coconut and ask you to determine the lowest floor from which if the coconut were to be thrown it definitely would break. Note that you get to use only one coconut, the building has 6 floors, and you must start from the ground moving upwards.
Using a technique called Boundary Value Analysis, we can decide that the valid test range for us is between the 1st floor to the 5th floor — the rationale here being that we surely know that the coconut won’t break from the ground floor and will surely break from the 6th floor. Now considering that we need to find the lowest floor from which the coconut can break, we can use another technique called Equivalence Partitioning to divide the 5 floors into 2 valid test sets — Floors 1 and 2 form the 1st set and floors 3 to 5 form the 2nd set. Let’s pick the 2nd floor to start. The coconut doesn’t break on being thrown from the 2nd floor. We now know that the coconut won’t break from the 1st and 2nd floors. We then proceed to the next set and pick the 3rd floor considering the floors above have a better probability to break the coconut and we want to find the lowest possible floor that will break the coconut. The coconut breaks on being thrown from the 3rd floor. We can safely assume it will surely break beyond the 3rd floor.
Using these techniques, we avoided trying from every floor and instead tried just 2 floors to find out the answer.
Similarly, while trying to enhance the quality of a product, the focus should be on effective testing and not exhaustive testing.
Principle 3 — Early testing
The third principle basically talks about the effectiveness of testing early. This is a no-brainer — early testing will help find issues faster. Testing shouldn’t be thought of as an afterthought or a supporting activity, and testing activities should be a part of the SDLC early on, right from the point of requirements analysis. Even early feedback from the testing team at this stage helps uncover scenarios/conditions that were either not thought through or not thought about at all. Correction at this stage itself will help save a lot of time, efforts, and cost rather than later on through actual testing where the cost and time of correction will escalate.
The image above (Source: NIST) clearly illustrates that a well-defined and understood requirement will help reduce the possibility of bugs later on in the next phases and consequently will help reduce the cost to fix them.
Principle 4 — Defect clustering
The Pareto principle, also known as the the 80/20 rule states that most things in life are not evenly distributed. Simply put — 80% of the outcomes result from 20% of all causes.
The fourth principle talks about this principle in the context of a software system. It can be deduced that 80% of the defects/bugs can be found in 20% of modules of the system.
This can be attributed to the fact that some areas of the system are tricky to develop and the code might be complex. A proper risk assessment should be done with feedback from the development team to determine such problem areas. Initial testing might seem proportionate to the coverage areas. Based on the defects/bugs found, a reassessment and redistribution of testing efforts should be done to focus on the 20% problem areas that cause 80% of the bugs.
Principle 5 — Pesticide paradox
The pesticide paradox basically states that pests and insects develop immunity to pesticides and insecticides with time and require more potent techniques to be dealt with.
The fifth principle talks about this paradox in the context of a software system. Running the same set of tests on a system will no longer help uncover any new bugs.
How do we overcome this paradox? Well, test engineers need to constantly upgrade their tests and suites to include new changes done in the system. These revised tests should be properly reviewed and fine-tuned. Similarly, outdated tests should be discontinued or archived.
Principle 6 — Testing is context dependent
The sixth principle talks about the fact that testing should have a context.
Every application or system is built with a purpose. The end-users of the system will use the system to fulfill a need. It is this context that should also play a part in determining the testing to be done for the system. A healthcare system differs from a shopping portal and the testing will also have to be different based on the expectations and criticality of the system. To add to this, there are also international standards defined to help determine that the software conforms to certain established benchmarks/criteria. For example, the ISO 27001 standard for information security systems differs from the ISO 9564 standard for e-commerce systems. Testing should importantly consider these standards.
Principle 7 — Absence of errors fallacy
The seventh principle states that the absence of error is a fallacy i.e. finding and fixing defects does not help if the system itself is unusable and does not fulfill the user’s expectations.
The objective of any system built is to fulfill a requirement of its end-users. It wouldn’t matter if the system has very few bugs or if all bugs have been fixed when the system itself has not been built based on the users’ requirements. A wrong requirement even though built and tested correctly will still result in an unwanted product or system. Testing should importantly validate that the system built is doing exactly what the user expects of it and satisfies the business needs.
The images above clearly illustrate the difference in requirements for a swing. A swing without a back support won’t satisfy a client who desires a swing with back support.
History is replete with examples of the dangers of deploying untested software. These seven fundamental principles of testing are indispensable if you wish to enhance your brand’s reputation, satisfy and retain clients, get higher returns, and ensure overall product success!
