Finding the right balance in exploratory testing
As a tester in an agile environment, exploratory testing is a quite familiar testing approach. When exploring, you execute as soon as you think of a test. This is one of the key attributes that distinguishes exploratory testing from scripted testing. As you explore, you discover how the software operates, you learn about its quirks and peculiarities. You watch carefully, looking for subtle clues about where there might be a nest of bugs lurking.
Focusing on the most important information to discover, is one of the core skills of a master explorer. It can be quite tricky and difficult. You will need to find the right balance in your exploratory testing between testing “too deep” or “too shallow”.
Think of it as adding the right amount of sugar to your coffee :) putting too much will make it too sweet, while too little will make it bitter (Yuk!).
Let’s examine it a bit further…
What does it actually mean testing “too deep”? You will find yourself raising bugs which are not “user focused”, many of them will be rejected by the product owner. Also, when you share your testing scenarios, your team will raise questions such as: “Why are you so focusing on that area? We have a great amount of automation tests coverage at that specific part, why don’t you put your effort in…”
Going “too shallow” is almost as letting your end user test for you, users don’t compromise on quality. They don’t care how many tests you ran on your application, they care about the product quality and that it answers their needs. You will also have many “escaped defects” originated from stakeholders, end users or from the company employees themselves. Your team might be wondering about the domain quality and might find themselves constantly adding scope to your tests scenarios.
Two Key Aspects
As I see it, there are 2 key indicators that can help you understand your position on the scale between deep testing and shallow testing:
Team feedback
By team I mean, product owners, developers and other QA partners that provide a constant feedback to your work. They are the ones that work closely to you and to the product, try to be open minded as you can to their advices. Their inputs are priceless and invaluable!
Production Status
Feel & know production quality status at any given time! Receiving many bugs from support team or from the end user, bugs you are not aware of, this means you had been testing too shallow. On the other hand, focusing on rare cases and getting your bugs pushed pack, might tell that you’ve tested too deep.
Balanced vs Edge Cases
Keeping it in the balanced point is possible but could also be risky. We tend to get used to it and might miss trying new testing approaches such as: Using testing personas, analyzing the data analytics and following the new user flows (since we are living in an agile environment — this can constantly change). Also, we might take testing on different platforms (IPad, mobile web, etc.) too obvious.
In some situations, we do swing between both sides: big new features, code refactoring or a huge effect on another site areas, can cause you to test too deeply. While, starting a new project and you are not so familiar with the business requirements, the exploratory testing world is new to you or you are just having a relaxed sprint with no big changes can cause you to test too shallow.
Guidelines
Consider using these guidelines next time to figure out when you’ve done enough exploration in a given area:
- Your exploration has answered all the open questions and there are no unknowns for you.
- You are not learning anything new. You have characterized the capabilities, limitations, and risks of the existing implementation. As a result, you feel like you are dealing with a known quantity with no more surprises to be uncovered, at least for now.
- More information will not change anything. You could gather more information, but there is no point.
- Focus your exploratory testing by using personas, testing tours, heuristics, user journeys or session-based test management. What you learn will help your team improve quality, not only in the impending release, but as the product evolves in the future.
Thanks,
Jasmine Shany.
3 highly recommended books:
