A guide to starting bug bashing now
— How to apply a process to bring organization and easy to the bug bash ceremony
Important: This is a step by step of how to organize a bug bash. More on the benefits of its use at QuintoAndar can be found in the article “How bug bashing frequently and consistently changed the team mindset towards quality and built connection at QuintoAndar”.
Introduction
Bug Bash is a test methodology, originally defined by Ron Patton on his book Software Testing in 2001. This approach to testing has been helpful on different squads at QuintoAndar and as a result, we have created a process to optimize its application.
Bug what?!
I first heard the term bug bash on my recruiting interview at QuintoAndar and it sounded good. Later, I was online searching for more information only to find out that this was a quite spread practice, used by many companies with one thing in common: an agile environment. Bug Bash is a test methodology originally defined by Ron Patton, 2001. In his book “Software testing” he describes bug bash as a procedure where all people involved in the product’s development cycle, put aside their regular day-to-day tasks and “pound on the product”. So, everybody who somehow has had a say on the product — from definition to design; from development to marketing — should be in the same room, testing the feature, in as many ways as possible before releasing it.
It brings the team — from stakeholders to development — closer together to cover as many test scenarios as possible and, therefore, reducing the chances of a defective release. Recognizing that this could be a viable and valuable alternative to our fast, intense pace of work, we started to slowly introduce the concept into the squad, always improving and adapting the approach according to the feature’s needs.
Heaven, is that you?
Sounds like a dream but if you have ever led a bug bash, you probably know how chaotic it can get to gather so many different profiles in a room: some people raising doubts about whether it is a bug or not, others asking where to report the findings, others having trouble accessing the system, everybody discussing how to follow up and prioritize bug fixes while testing and so on. Without a minimal prep, odds are that important details are going to be lost in between questions and post-its, people will get frustrated for not being listened to and, ultimately, what should be the most productive time finding bugs can turn out as a waste of time for the entire team.
In addition to the inherent complexity of coordinating a bug bash, we also had the challenge of applying the concept on a product that offers a complete business solution: from property searching to rental administration. That means back-end and front-end deliverables throughout the whole user experience. Therefore, here at QuintoAndar, it was imperative to have a flexible and adaptable process that could suit both scenarios.
We knew there was — and still is — a big challenge ahead of us: getting to know how to apply the bug bash effectively in different contexts. We needed a propeller. Meaning, a smooth process with essential and robust tools for centralization that was adaptable enough to be used on different kinds of deliveries’ validation.
So wondering about the above realizations, we’ve created a process focused on combining key points of testing along with key elements of bug reporting, as simplified as possible without compromising its efficiency.
Setting up a bug bash
As mentioned, the fundamentals for a good bug bash includes a prep phase in which the quality assurance person, sets up the conditions for testing, providing the temporary testers with tools that make their testing skills valued. There are three main steps to a good set up:
Step 1: Create a spreadsheet to centralize all findings
First things first: we need everybody on the same page. Literally! When trying to map problems, it is fundamental to report everything on one single document. This could mean a shared note, a text doc, a giant cardboard. We decided to use a spreadsheet in which we could also give some visual guidance of what was being tested. This choice was a collaboration with the team’s product designer who describes the benefits for the Design itself in the article “Product Design & Bug bash: how to validate interfaces and quality assure your delivery from end-to-end”.
A spreadsheet also allows the process to be easily automated to be smoother and more functional. In a bug bash spreadsheet, we have three tabs: Participants, Bug bash, Automatic report.
As QuintoAndar’s end-user applications are mostly web-based, testing needs to include different mobile and desktop devices. Sometimes the problem is device-related making its disclosure indispensable. So we created the “Participants” tab, in which each tester or participant of the bug bash inputs information such as name, simulated user, browser, device brand and device type. These are the basic information we need to start investigating — or fixing — the problem. Once that information is completed, we can carry on.
On the second tab “Bug bash” we combine the testing and bug reporting elements. An image of the context to be tested in the first column is followed by three columns where the tester inputs the tester’s name, a detailed description of the problem found as well as an image of it, if applicable. This way, if there are doubts about the report we know who to inquire about it and save time. This is particularly valuable when assuring front-end deliveries because we can validate the final product against the designed layout. If we are validating back-end deliveries, the front-end image is usually replaced with a workflow explaining what is happening on the back-end. This way, we keep people posted on the context even though is not visibly testable. People will test the product and report all problems, enhancements and doubts on this tab. When testing time is up, the team will go over the reported issues, reviewing and discussing what makes sense to be included in the fixing phase.
Due to this stage of severity evaluation and prioritization, the whole team will be aware of all issues found and select the ones related to that development cycle for reporting. This will optimize the reporting phase and avoid follow up meetings to prioritize the problems.
The third and last tab on the spreadsheet, named “Automatic report” is key to keep up with the speed of our processes. It contains a script that takes all the information from the last two tabs and automatically creates an issue report on our Jira board. This means, no information lost on the reporting process phase. This prevents a lot of reworking and makes the whole process way more efficient. The script is customizable so it is possible to add more fields to be reported.
Step 2: Create a context presentation
To be effective, the extent of the product to be tested should be very well defined to everybody involved in the testing process. The scope should be limited to the feature(s) added in that particular time window or development cycle. The clearer people are about the scope of that particular test ceremony, the less you will get to answer whether that is a bug or legacy problem. It is especially important to have this presentation in case stakeholders or squad outsiders are participating in the testing, cause they may not have the clarity that people inside the development itself have.
Step 3: Book a room and send invitation days ahead
Arrange a room to fit a large group of people while also minding people’s calendar is not always that simple. It could take a long time to match everybody’s calendar or have a big enough room available, so stay ahead. On the invitation, share the context presentation along with the bug bash spreadsheet and make sure to have given the permission needed to all participants. Another very important aspect of bug bashes is time. Make sure to spare at least 1h 30m ( one and a half hours) to invest in this ceremony. It may seem a lot but it is not when there are so many different people using the product and interacting to get a better approach to the user experience.
Leading a Bug Bash
Once the set up is completed, it is important to keep up the organized mood to create interaction as positive as possible. So we also have three important steps while leading the bug bash. For the best experience of the participants we recommend the following steps:
Step 1: Present the context and explain the bug reporting system
It is bug bash time! Start by presenting an agenda explaining how the event time will be divided. Make sure to allocate specific time slots for the context presentation, test and report, bug triage and prioritization. We understand multi-tasking is overrated — not to mention it is counterproductive. So by separating time slots, provides the peace of mind people need are to be more focused and engaged at each one of the bug bash stages.
Present the context, reminding all participants of the motivation behind that development cycle and the general technical aspects of it. Make it short and straightforward so it would not take longer than 10 minutes and everybody gets to put the hands-on quicker, allowing more time to what matters: testing!
Talking about testing, right after the contextual presentation, guide the testers on how the bug report will take place. Show them all the tools they will use, so everybody will work close to a standard.
Step 2: Keep calm and test on
Now that everybody knows what they need in order to test, everything is going to be perfect, right? Wrong.
It will take a while for people to get to the testing mood so do not get frustrated if the first 10–15 minutes are a bit chaotic. This is expected, especially if people are not acquainted with the process yet. After some time warming up, participants will fully understand the concept and be comfortable with the new tools they have. It tends to get smoother with time.
Also, encourage everybody to report every finding on the spreadsheet, even the ones they are not sure are bugs. This way, we can get a hint on the usability aspects and address non-functional issues earlier. Not only the code itself will be tested, but the user experience as well. For example, if more than two testers point something odd on a specific flow, even that is not a bug, it should be discussed to understand whether it should be changed before releasing the product.
Be available for answering doubts and prepared to tackle problems they may encounter using the tools.
Step 3: Triage and priority
Once the testing time is up, the whole group goes over the spreadsheets’ reports to evaluate bug severity and prioritize fix before launch. This is the time when everybody is listened to. We go over one by one of the reported problems and evaluate its impact on the release. If the team decides it is critical, we put a label to signal that. Once that is done, we automatically report all the issues and close the bug bash.
You are all set!
Now that you know the steps we use at QuintoAndar, you are ready to apply them on your own team or adapt the process to your reality. Because at the end of the day, the success of a bug bash depends on the dynamics your development’s environment requires. Keep an open mind when applying whatever version of a bug bash suits your team, cause although it may seem a significative work to be done, it really can change the mindset of the team as I talk a bit about here.
Always remember the key point of the bug bash: to guarantee the quality of the new features and make sure you can lean on a process to organize it with some simple and effective tools.
