Importance of Exploratory Testing
This story is co-authored together with Sivamoorthy Bose
Exploratory testing is an approach to testing whereby the testers dynamically design and execute tests based on their knowledge, exploration of the test item and the results of previous tests (according to ISTQB Glossary).
Exploratory testing is commonly confused with ad-hoc testing, which is performed on demand without scripted test cases, or monkey testing, which is done with random user actions. Exploratory testing is learning a product with small experiments and adapting new experiments during the testing session. Such kind of testing perfectly fits Agile with a similar mindset as Agile experiment.
Benefits of Exploratory Testing
- Exploratory testing catches more defects than automated.
Let’s see some of the metrics that show why exploratory testing is so important.
- 33 % of complex bugs in an application are found through exploratory testing.
- 29% of the UI Bugs are found through exploratory testing than automated script testing.
- Exploratory testing identifies 11% overall defects than automated script testing.
Source : https://www.perforce.com/blog/alm/what-is-exploratory-testing
Exploratory testing helps to deliver a more quality product to the market with more test coverage.
2. Exploratory Testing Helps to Share Domain Knowledge
Exploratory testing can be performed in pairs using peer programming principles. It assumes 2 roles: of driver and navigator. Driver is a person at the wheel, this person should focus on the application while he/she is performing actions and ask questions if they occur. The navigator is in an observer position and tells direction of further checks and makes notes of steps or findings. Those roles can be changed from time to time. This approach can not just help to find bugs, but also explore products and share knowledge of some features within pear.
3. Exploratory Testing Helps to Switch Focus of Attention
Human attention is an interesting mechanism. When we focus on something specific we don’t notice everything else unfocused. Even multitasking people don’t have more than one focus, they just switch attention very fast. Look at this video to see in-attentional blindness in action:
Confirmation bias is one more attentional tendency, which impacts testing. It is a tendency to pay attention to facts, which prove the personal beliefs. Let me share one of our experiences that we faced such bias during testing. We had a requirement of an expected colour scheme for the app describing first and secondary colour and where they should be shown. We almost skipped an important visual issue: write text was written on a light yellow button.
One of key benefits of exploratory testing is that it helps to deal with cognitive biases and focus on different perspectives in our application.
How to Transform to Exploratory Testing Mindset
This is an era, the entire QE industry is moving towards shift left testing and establishes QE process. As we focus on functional and automation testing but still we find gaps in test coverage. Every tester should have the mindset to do exploratory testing at least once in a week. There is always limitation in our automation scripts and it has boundaries. Automation regression testing alone cannot find bugs that cover all the scenarios. Scripted testing can only verify what is in the script, catching only conditions that are expected. Such tests can be a fine net that catches any bugs that try to get through it, but how do we know that the net covers all those scenarios?
Exploratory testing seeks to test the boundaries of the net, finding new behaviours that aren’t in any of the scripts.
Source : https://martinfowler.com/bliki/ExploratoryTesting.html
What attributes QA should have?
Inquiring — Exploratory testing doesn’t pre-determine what’s to be tested or how, so great exploratory testers are inquisitive and interested. They want to find out everything. They want to know what’s behind this door (or button) or and what’s hiding in this box (or feature). They need to be driven to learn more since they’re on their own when it comes to checking the different parts of the website or application and investigating its nooks and crannies.
Flexible — Great exploratory testers are competent test designers who can roll with the punches: if an investigation of a specific part of the software turns up an error, they need to be able to improvise new tests that look for related problems and issues. An exploratory tester who is good at coming up with impromptu additional tests can also determine the scope of a bug, or at least circumscribe the impact of it, which makes the task of fixing it easier for the developer.
Observant — Inquisitiveness isn’t enough on its own. Exploratory testers should not only be curious, but also able to notice any aberrations and document them. An excellent exploratory test requires the testers to observe settings, environmental factors, and internal mindsets as well.
Holistic — Exploratory testing needs to consider and record anything that may influence the testing. This could range from factors such as: wifi versus cellular, signal strength, battery level, or even ambient noise. The great exploratory tester embraces the need for additional investigation and fully considers all paths that should be investigated instead of settling for one scripted solution.
Excellent Communicator — Someone can be intelligent and inquisitive and awesome at finding defects, but it does little good if they can’t communicate their test findings to the developers. Without a separate test designer sitting down with the product manager to pre-determine individual testing steps and expected outcomes, testers are responsible for explaining the idea behind a given test, the circumstances surrounding it, and then the outcomes that result.
Source : https://test.io/exploratory-testing/
How to Organise Exploratory Testing Session
Even exploratory testing does not need scripted test cases, it needs some procedure. We recommend reserve time to perform exploratory testing on a regular basis. Session based approach is the best way to conduct this type of testing.
Prioritise the Bugs
During testing, each tester knows the cluster that has the most number of bugs in an application. Prioritise those clusters and start doing exploratory testing on those cluster code bases.
Include Exploratory Testing in the Test plan
When you create a master test plan, include exploratory testing as part of the plan. Have at least 1 tester to do this activity on a weekly basis.
- Include Module or feature to be focussed.
- Include What all the tool, technique, configuration.
Create a Time Bound for Testing
Have a specific time frame to do exploratory testing for each release and also validate those results with the development team and address them on time.
Testing Charters
There is a simple template suggested by Elisabeth Hendrickson in the book “Explore It!”, which can be used for planning the scope of exploratory testing sessions.
- Target: where will you explore? (feature, module).
- Resources: what should you use? (tool, technique, configuration).
- Information: what are you hoping to find?
There is a nice example provided by Elisabeth, which we especially like:
Testing Tours
Testing Tours is an amazing alternative to Charters. In the book “Exploratory Software Testing” James A. Whittaker suggests the concept of testing tours. The idea is to compare software with a city. QA is supposed to explore the city from different perspectives. You can select the existing tour or invent your own.
We suggest trying the garbage collector’s tour. In the real world the garbage collector spends a short time at each house, picking up the garbage. He has one specific task to do methodologically for each house. It can be used to test every menu item or all validation/ error messages, modal windows.
Personas
You may notice that you or your QA colleagues use the argument “As a user I expect….”. In many cases it is a valid expression and the QA person is really a user of the system being tested. QAs also have bank accounts, book flights or order food. But in many cases we are not users of the system, which we test. We can test incident management systems for oil production and most of us never worked on such dangerous productions. That is why it is important to know who are real end users and be flexible enough to switch the mindset. Persona enables QAs to test applications with fresh perspective.
You can define persona like here:
Janet is a 67 years old pensioner, who has a dog. Modern technologies are not important to her therefore she has a slow Wi-Fi connection at home, light version of Android tablet and no notebook. Test ordering dog accessories as Janet, who has plenty of free time and patience with her slow connection.
When QA assumes Janet’s persona, she would start using applications in a special way, not like ordinary. Her technical set-up may cause that long registration form not saved and more steps with saving data could be helpful. She may face issues with website responsiveness on the mobile device or can be confused by lack of loaders.
Conclusion
As we move on towards shift left testing and identify the critical bugs as early as possible then exploratory testing should be part of the testing life cycle. This also helps to identify the gaps in the test and automation coverage. Exploratory testing is nothing but playing around the application with no boundaries and rules. Lets enjoy doing it more often!
