Exploratory or “go with your guts” testing

When I joined Limehome, the team was about to release the product in less than one month and my challenge was to support the team testing the whole product. So what I did was just exploratory testing and I think the team would admit that the result was good.

What is exploratory testing?

Exploratory testing is a term coined by Cem Kaner in the 80’ and James Bach used it later. “Exploratory testing is simultaneous learning, test design, and test execution.”

Bach believes “testing is an exploratory process. It’s not just sometimes exploratory; it is inherently exploratory. In other words, whatever else you may be doing when testing at a professional level, you are also actively learning and making new choices about what to do next based on what you learn.”

There is a myth that before testing, you should create all the necessary test cases, and while testing, execute them. In this case, the tester is just an executor and there is no thinking or human intelligence involved. On the contrary, exploratory testing is a simultaneous process of test design and test execution.

In my experience, people never mention exploratory testing, even in the interviews they usually don’t ask if you know what exploratory testing is or if you know how to do it. But if we don’t say it’s more important than other kinds of testing, it’s as important as others. As a result, the tester should always learn about the product very deeply.

As Bach says, “no testing is completely exploratory, nor completely scripted. It’s a mix. Good testers learn to control the mix.”

Bach compares ET to a jigsaw puzzle: “When I solve a jigsaw puzzle, I change how I work as I learn about the puzzle and see the picture form. If I notice a big blotch of color, I might decide to collect all the pieces of that approximate color into one pile. If I notice some pieces with a particularly distinctive shape, I might collect those together. Sometimes I feel like I’m too disorganized, and when that happens, I can step back, analyze the situation, and adopt a more specific plan of attack. Notice how the process flows, and how it remains continuously, each moment, under the control of the practitioner. Isn’t this very much like the way you would assemble a jigsaw, too?”

Why do exploratory testing?

In ET, the tester has more freedom to discover the product and use their creativity to design test cases while exploring. The process is not predictable and that is what’s special about it. The tester explores the product gradually and asks many questions from either themselves or others to have a better understanding of the product. While some people, especially the ones outside the testing area, believe that testing should have a predefined structure, it has been proven the more the tester does exploratory testing, the more out-of-the-box bugs are exposed.

As Cem Kaner says: “No matter how many test cases of how many types you’ve created, you will run out of formally planned tests. You can keep testing. Run new tests as you think of them, without spending much time preparing or explaining the tests. Trust your instincts.”

The difference between scripted testing and exploratory testing

One big difference between exploratory testing and scripted testing is that the tester has more freedom to design the test cases and execute them in parallel, but in scripted testing, the tester is just checking the test cases that have been defined in advance.

As Bach mentioned, another difference is that scripted testing is based on explicit knowledge, something that you can write down or talk about but ET is based on tacit knowledge when you know how to do something but you can’t put it into words. It’s according to your knowledge; you’ve got it in your mind but sometimes it’s difficult to explain or describe.

And the main difference is thinking while testing. A tester is not like a machine that checks if some written scripts are working as expected or not. What Bach is trying to say is that the main difference between humans and machines is “thinking”, so there is some kind of testing that just humans are able to do because they can think while testing.

How to do exploratory testing

ET can be done in sessions as they’re called session-based test management. Each session is a limited period of time and has a specific mission. These are session-based testing sections:

• Charter or mission of the session
• Tester name
• Date and time
• List of bugs
• Test notes

After each session, there is a debriefing session where they discuss the issues during the session, provide feedback and decide if they need more sessions for testing. In the end, they prepare a report of the session.

Does exploratory testing fit in an agile context?

As Elizabeth Hendrickson mentioned in her book, “ET is a core testing practice for Agile teams.” Both automation and manual testing are needed in an agile environment. Automation scripts are used for regression testing but it’s not sufficient; an Agile team would also need manual testing. In each iteration, QA defines a charter and the scope of testing, does the exploratory testing, and prepares a report that includes videos or screenshots.

Takeaway

• Add it to the release process, even if exploratory tests are not sufficient in the testing process, they are indispensable
• Investing time to develop this mindset even when you are conducting a formal testing process. It’s as simple and evident as this; using your previous knowledge while testing

References:
https://www.satisfice.com/exploratory-testing
https://www.satisfice.com/download/session-based-test-management
Exploratory Testing in an Agile Context by Elisabeth Hendrickson
Testing Computer Software by Kaner