ISTQB — CTFL - Chapter I: Fundamentals of Testing II
The second part of Chapter I from ISTQB-CTFL consists of 1.3 Seven Testing Principles, 1.4 Test Process & 1.5 The Psychology of Testing.
Every technology really needs to be shipped with a special manual — not how to use it but why, when and for what
- Alan Kay
1.3 Seven Testing Principles
For the last 50 years, a variety of principles are suggested in the area of software testing as more of a guideline. Here are the 7 testing principles that take place in the ISTQB:
- Testing shows the presence of defects, not their absence
When we perform a test, it decreases the chance of having undiscovered defects, but it does not guarantee the absence of defects. Testing can only confirm their existence, but it cannot guarantee that the software is entirely free of defects as some may go unnoticed. - Exhaustive testing is impossible
Think of a scenario where there are five input fields each can have 10 different entries/data. In order to test all the combinations (5^10) we have to spend an unrealistic amount of resources. It is not feasible to test all cases (except for the trivial cases) but rather we should focus on risk analysis, priorities, and test techniques. - Early testing saves time and money
Discovering a defect in the early stages of the SDLC can save time, money and effort that may otherwise be spent more in later stages, such as during production which can be quite expensive. We can make use of static and dynamic testing as soon as possible. For example, reviewing the requirement analysis prior to the developer’s implementation or running and testing the code once it's ready, can prove advantageous. - Defects cluster together
After observing the characteristics of defects, it becomes clear that the majority of bugs are found in the same group of modules, which can be thought of as clusters. One possible explanation for this is that there may be a complex module or package that contains multiple defects. It is crucial to prioritize the results of risk analysis and test efforts, as emphasized in the second principle. The Pareto Principle, also known as the 80/20 rule, states that 80% of the outcomes are the result of 20% of the effort. In other words, 20% of the package or module is responsible for 80% of the defects. - Beware of the pesticide paradox
When we repeat the same test cases/scenarios again and again without altering any part according to the new information then these tests will no longer find any new defects because they are now obsolete. This notion is a metaphor for the case where pesticides are no longer effective at killing insects after a while because they get used to them and grow immune to them. - Testing is context dependent
It is important to create distinct test cases for different conditions and contexts. What we test and how we test vary depending on the situation. As the project's scope, quality, and budget change, testing must adapt to these changes. For instance, the testing methods employed in safety-critical industries, such as aeroplanes and military software, differ significantly from those used in e-commerce software. - Absence-of-errors is a fallacy
It is a common misconception that simply identifying and correcting numerous defects will guarantee the success of a company or its software. The success of the software is determined by various factors, and fixing bugs alone is not enough. In light of the first and second principles, it is incorrect to assume that testers can execute all possible tests and discover every defect.
1.4 Test Process
This part will discuss the steps that the testers can take through testing the object. However, keep in mind that there is no universal software test process, rather we have commonly accepted procedures, techniques and activities which will increase the chance of achieving the goals of testing. It means that we can remove or add some steps along the way.
Test Process in Context
Many factors affect how the test process is shaped (but not limited to those mentioned below).
— The SDLC model and project methodologies that are used (e.g. working in an Agile company requires the test process to be agile and effective as well)
— Having measurable coverage criteria for the test basis is crucial since it allows testers to show the accomplishment of the test objectives
— Product and project risks, business domain, operational constraints, policies and regulations also have a big impact
— Test products, required internal and external standards and activities define how to manage and arrange the test process
Test Process Activities
There are a couple of activities mentioned in the ISTQB — CTFL Syllabus. Some of them seem logically sequential but most of the time they are implemented iteratively (there is a type of overlapping). Because nowadays we have many Agile companies that use small iterations to decide, design, build and test the software continuously, e.g. tester might write a test case for one story and execute some other test cases for another story.
- Test Planning
We begin with the first step of a multi-step process to establish testing objectives and determine the approach for each test object to meet requirements. It's important to note that this plan will be continually updated throughout the cycles based on feedback, despite being created at the beginning. It's also important to differentiate between the project plan and the test plan, with the latter typically being the responsibility of the test leader/manager. - Test Monitoring & Control
It's important to note that monitoring progress is an ongoing task that requires constant attention and take action accordingly. We have specific metrics to follow to meet the test plan objectives. Typically, we have exit criteria (similar to acceptance criteria in agile methodology) and it's our responsibility to meet them. To demonstrate the test results against the coverage criteria, we provide a Test Progress Report. - Test Analysis
During this step, we analyze the test basis and determine the necessary test conditions, which outline what needs to be tested. This involves examining various objects, such as requirement specifications, business rules, use cases, implementation of the component/system, interface, design, risk analysis reports, and more. We can also identify any tasks that are relevant to the given requirements. Additionally, we may not outline all the steps for trivial test cases but instead, provide a test condition that must be met. It’s essential to specify these conditions during this step. - Test Design
After deciding what to test, the next step is to determine how to test it, which is outlined in the test design phase. During this process, we define and prioritize test cases, specify the required test data (some may not be necessary), design the test environment (including software, network, and hardware), and establish the bi-directional traceability between the test basis and test conditions. - Test Implementation
We determine if a product is ready for testing and prioritise test procedures. We then create a collection of test cases called test suites and organize them for the testing schedule. Additionally, we prepare the testing environment that was previously designed and ensure that the test data is correctly loaded into it. It is important to note that this step is mostly theoretical, and we may create empty test suites and cases that we will fill in later during the process. - Test Execution
During this step, we run/execute the test suites (actually each test case/scenario) that were previously prepared and compare the obtained results with the expected results based on the acceptance and exit criteria. In manual testing, we follow each step and document any notable findings, such as IDs, test items, tools, and test ware. Automated testing involves executing the test and allowing the steps to run automatically, while also recording the process. The test outcomes are then documented as pass, fail, blocked, etc., and any observed defects are reported. - Test Completion
As we wrap up our testing activities, it's important to gather data on the number of tests conducted, defects found and reported, and their priority and severity levels. Test completion typically occurs at a project milestone, such as a software release or the end of a sprint (an Agile term which indicates a period for a process, typically 2 weeks). We must ensure that all defects are properly reported and create a test summary to share with stakeholders, including team leaders, product owners or project managers. It's worth mentioning that the test summary report differs from the progress report, which the later generated during the testing process but summary reports are created at the end of a test plan. By gathering lessons learned, we can improve both the quality of the software and the overall testing process.
Test Work Products
There are 7 test process activities and in each step, we have different products as a result of these steps (like an outcome of what we do at each activity). Since the test process might differ from project to project or company to company, there is a variation in the work products as well.
- Test Planning → Test Plan
- Test Monitoring → Test Progress/Summary Report
- Test Analysis → Test Conditions and Test Charters
- Test Design → Test Cases and Test Data
- Test Implementation → Test & Test Suites, Test Execution Schedule
- Test Execution → Test Case Status and Defect Reports
- Test Completion → Test Summary Report and Lessons Learned Report
1.5 The Psychology of Testing
Since we deal with human beings at all steps in the development and testing stages (and overall, obviously), we should know about the nature of what we get into.
As testers, finding defects might be perceived as criticism by others and while communicating with the developer or product owner, this point should be kept in mind. There are two terminologies to consider while doing so:
- Confirmation Bias → It is about the difficulty accepting the fact that something is wrong, e.g. the developer might believe that the code is fine even though the tester later shows that it does not work as it’s supposed to
- Cognitive Bias → This one is about not having related background, information or experience to understand or accept the outcomes from the testing process (i.e. defects, failures)
Therefore, communication skills are one the most important skills that a test engineer should have because, in the end, that person is the one who gives the bad news to the developer, team leader or PM.
— Start with collaboration. Remind everyone of the common goal
— Emphasize the benefits of testing because it might be perceived as a destructive activity
— Communicate test results in a neutral & objective way, nothing is personal
— Try to understand the other person’s feelings, and confirm that the other person is understood the point clearly (objectively)
Tester’s and Developer’s Mindsets
It is widely recognized that developers and testers have distinct ways of thinking. Developers aim to create, construct and innovate based on the requirements given to them, while testers focus on verifying, validating and identifying any issues following established standards. Typically, developers are more invested in designing and building solutions rather than in contemplating what might be wrong with those solutions. Testers, on the other hand, tend to be more curious, detail-oriented and approach their work with a healthy dose of professional scepticism.
On the other hand, having an independent tester or test team might bring a different perspective. However, having an independent tester in an Agile company would not be beneficial since the tester is a part of the team, i.e. developers, business analysts, product owners and testers.