Increasing Quality Efficiency with Exploratory Testing
Explore and break your own product — Guide for better testing coverage
“Perfection has to do with the end product, but excellence has to do with the process” — Jerry Moran
In this article, I’ll be talking about how my team changed the way we do exploratory testing to improve product quality. Before going into the details of it, let me first explain you what exactly exploratory testing is.
Exploratory testing is a type of testing in which there are no initial scripted tests written, but the testing is based on the thought process on the fly. It is mainly a ‘thinking’ activity. It is the key to recognizing gaps and making changes in the product for improving its overall design, performance, and functioning without the need to undergo a long and planned testing approach.
Thumb rule for exploratory testing: minimum planning and maximum execution. The more you explore together, the better you work as a team. Exploratory testing is not just about finding bugs —it’s a way to add a shield in the process of quality over your product before it reaches customers. We have to think about engineering excellence while reducing errors in the product.
Let’s give humans some value over machines. We could decrease the testing time by automating more tests and filling out the gaps, but we still need humans for catching these in and out issues.
Initially, we were performing “exploratory testing” in a different way, testing the feature that was being released in that version of the app. However, it was not effective as we had hoped. We continued to find issues in production. I was tasked with driving the efforts in improving exploratory testing.
As a quality lead of the team, I made some changes in the process, worked around, did some analysis, and adapted a new approach to exploratory testing. The main target with this new approach was not just the feature being added in that particular release, but also some other main modules where we were getting escalations. We started with pairing each author with a helper, whose main focus was on breaking the product as much as they can without just relying on the scripted test cases like we do in regression.
The main criteria that our team adopted was hit it hard, be crazy, and break the product in every possible angle you can.
Framework we used for “New Approach”
Exploratory Framework
How can you get started?
You would need the tools and techniques mentioned below to get started with the process.
Testing Environments
Environment plays the foundational role that you are going to use for testing. Internal testing environment rarely mimic what customer use. Hence it is important to tests against an environment which includes actual customer use cases as well some involving different combinations of data instead of direct feature use cases.
Configuration
Figure out all the different configurations the customer would be using. There could be combination of different features, new operating system version in the market or a new device model. Once the data is collected, setting up an environment with the configuration to be used for testing would be ideal.
User Personas
Putting yourself in the shoes of the end-user or the admin and then trying to analyze and test the flow will make a huge difference in the testing. Giving up your knowledge for those 2 hours and try to determine the use case a customer will execute. Think it from the customer perspective.
Charter
Laying down the specific areas that need to be focused on during testing for a particular release. Rotating different modules per release will help the team gain their knowledge and also make sure that the feature is tested well. For example, if a person wants to learn about any existing feature in the product they can choose that because with exploratory it will help them increase their knowledge in that domain as well.
Who can participate
Team members (including DevOps engineers as well as developers), Product Managers, UI/UX team, Integration teams (teams getting affected by some of your modules). People from different team often look the product from a different perspective.
Scope
Priority is given to the features in scope per release but it is open-ended for anyone to pick any module; modules that were most affected and have customer escalations
Time Limit
Making testing time-specific gets your entire focus on one particular task instead of multi-tasking. We started by blocking two hours per release where everyone gets on a Zoom call so that if anyone had any questions, they could ask right away instead of chasing them over slack. With some changes in the future releases, we are also thinking about splitting the two hours over two days so that testing does not eat up too much of one day.
Tactics
Identify the type of specification for the feature/module and risk a new feature could have on the stakeholders; which type of testing to be performed, and which entry and exit criteria apply (like devices and the OS versions required for a feature). The more widely testing is done for any feature, the lower the chances of getting any customer-found bugs.
Document and Process
Documenting the steps so that it can be used for introspection, future improvements and determining the effectiveness of this approach.
Advantages of Exploratory Testing
- More quality-driven product
- Intelligent testing more than scripted testing
- Important bugs are found
- Team members get to know more about the product
Future improvements in the framework
- Including other team members like UI/UX teams and PM
- Save the Date invite ahead of time to block everyone’s calendars
- 1 hour each on two different days to accommodate everyone’s schedule
- RCCA on bugs found
Conclusion
Without doubt, exploratory testing brings out the creativity of each person and has an adequate effect on improving the quality of the product. This technique works well because no two sets of eyes can have the same perspective in life; our team’s results speak for themselves.
Hopefully, with the results it can encourage other teams to start using this approach; it takes somewhat more time and effort, but it’s far more effective from a product quality standpoint.
