How We Use Exploratory Testing Sessions to Prevent Bugs
Shipping quality software is one of the primary objectives for engineering teams. We’ve been able to successfully reduce our defect escapement rate at Vivid Seats by facilitating Exploratory Testing Sessions prior to releasing a feature. We encourage every member of our delivery teams to embrace this process, with guidance led by our test engineers.
Exploratory Testing is a practice by which a diverse group of participants on and off the team run through scenarios of a feature prior to release in order to find bugs or defects that may have not otherwise been found. This practice regularly includes off-team participants to create a holistic review of the feature. We also include delivery team members to encourage ownership of testing and quality, not just the test engineer.
The Exploratory Testing Session
At Vivid Seats, we design our exploratory sessions to be approximately one hour in length. Before the session, the test engineer assigned to the team or feature will outline specific key scenarios for the feature. The test engineer will also create a document to track any defects found during the session. It’s important to share this document with the participants in the session prior to the start of the session. One can use a tool such as Google Sheets for this purpose, as it is collaborative and easily shared. Other collaboration tools like Trello or Office 365 can work as well.
Rather than instruct the testers to run through specific flows, the test engineer will create a testing charter. A testing charter will outline an area of focus, but allow for the individuals testing to explore on their own. This method helps avoid any biased results in the testing session from explicitly instructing testers to test specific flows. In practice, this allows more input from the end user perspective and allows the tester to provide feedback on the feature from a functional and design perspective.
When to hold an Exploratory Testing Session
We aim to host our exploratory sessions when the feature is at completion or near-completion state. If you host the session too early in the delivery process, the defects found will be the trivial ones that would have otherwise been found. If you host the session too late, say after shipping the feature, then the defects found have already escaped to production. For larger features, it may be possible to carve out specific sections of functionality that can be tested in isolation. In these scenarios, you may string together several exploratory sessions to test different areas of a feature.
Who to invite to an Exploratory Testing Session
We intentionally invite participants from outside the delivery team. It is important to invite participants with a diverse set of skills, such as designers, product owners, or business stakeholders. It is also crucial to communicate up front to these participants the state of the feature they are using in this session. Be clear: it is not yet ready to go, it may contain defects, and the goal of the session is to identify any issues prior to release. This will set expectations so that a stakeholder can participate, but not assume the feature is finished at this point.
Was this information helpful?
We hope sharing why and how we run Exploratory Testing Sessions at Vivid Seats can be helpful to others. Special thanks to a handful of our engineering team members for assisting with this article, namely Nick Fagen.
Additionally, we are actively hiring for several roles on our engineering team. If you’re interested to learn more about open positions, check out our careers page at vividseats.com/careers for more information.
