Test Bash: Organising release testing the exploratory way

Jyoti Patel
Jul 28 · 6 min read

Written by Ladislav Cicon & Jyoti Patel

Have you ever been part of a project so complex, that the only time it all clicked together was after more than a years worth of effort?

Stories were delivered in a DevOps manner, so everything was tested and delivered into production almost instantly, though not much of it is used yet. You are about to approach your beta customers and you are unsure. How does it perform from an end-user perspective in the production environment? Does it handle all the use cases well? What was that first use case?!

We wanted to be sure and that’s how our version of Test Bash was formed. We combined a bit of exploratory testing, a bit of test planning, and a whole lot of experience to create our way of release testing for big projects.

Test Bash

Image for post
Image for post

What does the term “Test Bash’’ mean? The Ministry of Testing uses the term “TestBash” as a name for the great conferences they organise. We use it internally as a way of defining a time-boxed effort towards testing a product from the user perspective via exploratory testing.

At Wandera, this consisted of QA Engineers from different agile teams coming together to assess the quality of our latest and so far biggest project. We sat together in one room and worked in a way that is not usually seen in scrum teams, for an entire day as a QA collective. This is how we did it, but your team could include developers, products owners, or any other stakeholders as well.

Why do it?

Continuous delivery does well to reduce the risk of causing issues in production by testing and releasing in pieces. But what if small parts of functionality are coming together to meet a larger objective? Agile teams work hard to ensure their code works correctly; but what about integrating with other team’s work to reach a goal that has been in motion for months and only makes sense for users all together? Developing and testing in small increments sometimes stops you from seeing the forest from the trees, especially after months in the project.

The main goal of the Test Bash is to take a fresh look at the product; ideally in a production environment where all the pieces are enabled and integrated together from an end-user perspective.

What’s important for a successful Test Bash

Before the Test Bash was born in Wandera we tried several different approaches in big, impactful projects. Sadly we suffered from running over budget in both resources and time.

That’s when we began to brainstorm. We use exploratory testing on an incremental level, but why not use this approach when you are exploring the project as a whole?

In order to do this successfully, we found that there needs to be some structure when doing it for big projects. There should be a plan to share with relevant people; but at the same time it needs to be done in a way to not jeopardise the benefits of exploratory testing. It should be quick test design, where the focus is exploring rather following scripts. We did this by using exploratory test charters.

Exploratory testing with Charters

Test charters were prepared, reviewed and read by everyone involved before the Test Bash. The product owner overseeing the project also reviewed these, to ensure that customer and product expectations were aligned with the engineering team. This means you need to set aside time to do the test design prior to the Test Bash itself.

These charters defined a goal to focus on during each time-boxed exploratory session. The time-box is essential, as it helps define the boundary of time to explore before needing to stop and move on. We also added a capabilities matrix, which is a list of functionalities assigned to each exploratory charter. This helped us avoid duplication of effort and helped our product owner with an overview.

Image for post
Image for post
Example test charter used

Documentation of issues

With such a big feature, it was inevitable that there were going to be bugs and limitations found during the session. We wanted a go-to page where all issues found could be captured. We use JIRA and Confluence in the company, and found the following worked best for us to organise issues:

  • A JIRA label that was appended to all bugs raised (all bugs were to be documented).
  • A Confluence page displaying bugs, which matched this label.
  • The same Confluence page with a limitations table. This is to capture more abstract issues of limitations, that can’t really be captured in a concrete bug ticket.
Image for post
Image for post
Limitations table

Preparation and Organisation

We have learnt over time that good organisation is key for any event involving more than one person. You need to pay attention to the organisation of your Test Bash, and you’ll get effective time spent in exchange.

In our case, it was making sure the product owner and other QA Engineers really reviewed exploratory charters and the depth these charters went into. It’s essential to define the scope for your day.

You also want to work without interruptions. So book a meeting room, book yourself off in the calendar for a whole day. No, and I mean no meetings, not even stand up. Be ready to test all day long. We couldn’t imagine a better day. Book your product owner or stakeholders to discuss the results at the end of the day as well.

Then there are things which are low in Maslow’s hierarchy of needs but fuels you through the event. Have snacks, water, coffee, pizza and testing devices with cables and chargers ready. This will prevent tons of small issues from interrupting the Test Bash.

Morning Kickoff

At the start of the day do a short intro to the Test Bash. Explain the main business case, and pick the charters that everyone will start with. Don’t forget to repeat the rules and the schedule of the day.

Test Bash

With all the preparation you’ve done the Test Bash should be a delight. However don’t forget to review the efforts during the day and keep up with time-boxes in your charters. Since you are testing with colleagues, try to swap areas so you all are testing in functionalities that aren’t familiar territory.

Take pauses to refresh your mind, reach out for your colleagues if something is suspicious, and speak up if you need help. Anything that assists you in being more effective is allowed. Most importantly, don’t forget to enjoy it!

Debrief with stakeholders

As mentioned before, look through what you have discovered in the day with the stakeholders and present the findings; especially your overall feelings about the feature. It can be that it’s a great piece of technology without functional bugs but you suffered all day long with setting it up. You don’t want your users to suffer, and it won’t help the business.

Assess the priority of every bug you have created or limitation you have hit with the product owner, so the results are taken into the account immediately and planned accordingly.

Resulting Outcome

In the end, our Test Bash went successfully! Six QA Engineers came together for the day to assess the quality of a year-long project. We had a great time working together in one room assessing what we have been working on separately in our teams.

Image for post
Image for post

We logged 22 bugs ranging from usability issues to major problems, as well as a few limitations which were prioritised together with the product owner. We also prepared an executive summary for stakeholders of the project so no one was left uninformed.

A Retrospective was held later, and it revealed that this was by far the best approach to testing a big feature we have tried in Wandera or elsewhere. It’s important to say that teams have been testing the feature for months and Test Bash isn’t a magical day to cover your testing needs. It really is the icing on the cake of year longs worth of effort.

So tweak it for your company needs, give it a try, and let us know what worked for you. We were inspired to try this from a testing conference, so let’s build on this inspiration chain and make it even better!

Wandera Engineering

Insights. Ideas. Concepts. Articles. Codes.

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch

Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore

Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store