What is Exploratory Testing?
It has been 34 years since Cem Kaner coined the term to describe a style of skilled multidisciplinary testing common in Silicon Valley. I’ve walked the path of exploratory testing for 25 years and it has been a foundational practice in becoming the testing professional I am today. Let’s look at what it is, to understand why it still matters — more than ever.
An Approach to Testing
Exploratory Testing is an approach to testing that centers the person doing testing by emphasizing intertwined test design and execution with continuous learning where next test is influenced by lessons on previous tests. As an approach, it gives a frame on how we do testing in a skilled way.
We use and grow multidisciplinary knowledge for fuller picture of empirical information testing can provide. With the product as our external imagination, we are grounded on what is there but inspired to seek beyond it.
We learn with every test about the application under test, ourselves as the tool doing the testing, other tools helpful in extending our capabilities and the helpful ways we can view the world the application lives in.
We keep track of testing that has been done, needs doing and how this all integrates with the rest of the people working on similar or interconnected themes.
What makes our activity exploratory testing over other exploratory approaches founded in curiosity is the intent to evaluate. We evaluate, seek for information we are missing, making sure what we know is real with empirical evidence.
Specifying It By Contrast
It’s not a surprise that some folks would like to call exploratory testing just testing. In many ways it is the only way of testing that makes sense — incorporating active learning is central to our success with software these days.
To contrast exploratory testing with what people often refer to as manual testing, exploratory testing as a skilled approach encompasses use of programming for testing purposes. We use our brains, our hands as well as programs to dig in deep while testing. Sometimes the way of including test automation happens through means of collaboration, where ideas from exploratory testing drive implementation of automation that makes sense.
To contrast exploratory testing with people refer to as scripted testing, exploratory testing isn’t driven by scripts. If we create scripts from exploratory testing, we know to use them in exploratory fashion remembering that active thinking should always be present even when the script supports us in remembering a basic flow. We’ve talked a lot about scripted testing as an approach where we separate design — deciding what to test, and execution — making the test happen, and thus lowering our chances of active learning targeting the most recent understanding of the risk profile.
Another contrast to programming centric views to testing comes with embracing the multidisciplinary view of of testing where asking questions like “is my application breaking the law today after recent changes?” is something routinely encoded into exploratory testing, but often out of scope for a group of automators.
The Three Scopes of Exploratory Testing
As an approach, we can time and integrate this into our development efforts in many ways.
I find there are three ways of scoping:
- exploratory testing as a way of extending existing test cases
- exploratory testing as a limited timebox
- exploratory testing as frame of all thinking
When exploratory testing is scoped as a way of extending existing test cases, the way of working does not look very exploratory. In places like this, you find yourself wondering why Tina is so successful with the same tests Toni is missing problems with. The secret is that one of them actively extends how they understand the test cases, refuses to follow exact instructions or only instructions. They learn and find new perspectives.
When exploratory testing is scoped as a limited time box, you find people feeling they can only be free on Friday afternoons. These are moments in project life that are structured separately, with different expectations of how things flow and where focus is. This is a great way of scoping exploratory testing into a process where the natural inclination is to believe we know where the tasks begin and end, and need explicit encouragement to see if what holds true.
When exploratory testing is the frame of all thinking, it encompasses all considerations about testing. We start with exploratory testing to identify something that needs documenting, and take time to document it — perhaps with automation that is the modern way of documenting as executable specifications.
Listen to Language
If you hear: “My boss asked me to test search so I searched something that was found and something that wasn’t and reported I was done” you may be witnessing very low quality exploratory testing. It relies on following high-level orders to an extent the tester can imagine based on their knowledge.
If you hear: “My colleague wrote me 50 cases of what to try out with search and I tried them and reported I was done” you may be witnessing testing that isn’t exploratory. There is no hint of learning, and owning responsibility of quality of the testing that happens.
If you hear: “My boss asked me to test search so I came back with 50 quick ideas of what could be relevant and we figured out I’d just do 10 of them before we decided if going further was worthwhile“, you are likely to be witnessing exploratory testing.
Similarly, in the automation first space, if you hear: “I got a Jira ticket saying I should automate this. I did, and found some problems while at it, and extended existing automation because of the stuff I learned.”, you may be seeing someone who is exploratory testing.
If you hear: “The Jira ticket said to automate A, B, and C, and I automated A and B, C could not be automated.”, you may be witnessing testing that isn’t exploratory.
Look at who is in the center: is the tester doing the work and learning actively applying a better way of doing the overall testing? If yes, that is exploratory testing.
As skilled approach, it is only as good as the skill of the person applying it. With focus on learning through, skill may be a problem of today, but improved upon every day as exploratory testing is being done. If you find yourself not learning, you most likely are not exploring. With the product as your external imagination, you should find yourself imagining new routes through the functionalities, new users with new perspectives, and relevant information your project teams would be happy to make justified decisions on their take for the risk. With and without automation, in a good balance.
