Doing guerilla tests like Tarantino

How Covid-19 (and Tarantino) helped us scale guerilla tests at BlaBlaCar

A few months ago, we implemented a new method to improve guerilla tests (a research technique where you go to a public space and gather quick user feedback on your design ideas), adapting them to a remote context. During our last “remote guerilla test”, we came across this quote from a user:

“I am feeling super great because BlaBlaCar’s admin is a mother****** 🤡😠”.

This quote seems to be taken from a Quentin Tarantino movie, right? Apart from the fact that I would have loved to dig deeper into why the user said this, I started thinking about how this new technique shares some similarities with the filmmaker.

1/How did we end up referring to Tarantino?

Here’s the story: When I joined BlaBlaCar in September 2020, one of the first things I did was a few workshops with Designers and Product Managers to identify their user research needs in our Product and Design Framework. This framework represents the process with specific stages every product team uses to tackle a problem; quite similar to the user-centered design framework.

Designers and PMs were asked 2 things: 1/to think about “A time (or several times) when you wanted to learn from users but did not do it and why you did not do it” and 2/to prioritise “What will be a game changer in your daily life?”

At the end of the workshops, I had a good vision of where we already performed well (discovery and usability tests mainly) and where we needed to put some more effort. The main uncovered need was to have rapid feedback from users during design phases (ideation phases) to help teams converge. This could be showing a design, a low-fidelity flow or different variations of design/content. This need used to be covered with guerrilla tests: going on the street or to a coffee shop and testing it with random people. But with the pandemic in full swing, product teams were at home and unable to do these kinds of tests anymore. And even before Covid, guerilla tests showed some limitations: they were mostly done in France, and especially in Paris, when more than 80% of our users are outside France. Also, for most projects, we need our users’ voice rather than collecting feedback from random people.

This was especially crucial for content in our product:

This is why we decided to tackle the following challenge: empower product teams with better guerilla tests, making it possible to collect user feedback (and non-users if needed) in 24–48h, in every country we are operating in and with low additional budget. We identified 2 pilot projects and a first use case: content in the product.

Tarantino’s 1st tip: Seamless execution

Here I am not talking about the fact that in Tarantino’s movies you have a lot of (bloody) executions, but more on the similarity between a “guerilla test” and a “guerilla movie”. They are both characterized by being agile, low budget and freeing themselves from constraints. For a guerilla movie, it could be filming without authorization and for guerilla tests, going on the street and asking people for their feedback. The legend says that a lot of the filming of Tarantino’s first movie, Reservoir Dogs, was made “guerilla style” and most of the action takes place in a single warehouse. This did not stop it from becoming one of his greatest films. Just as guerilla tests can provide very valuable feedback to product teams.

To start, we listed some KPI we wanted to reach:

• Reach at least 10 users (typically in a guerilla tests you reach 5–6 users)
• The results influence decision-making (moving forward with one solution or keep iterating or perform a different type of testing)
• Collect the responses within 2–3 days maximum

Here is how we executed it:

Step 1: Find the best way to reach our (international) users

At BlaBlaCar we are lucky to have a caring community of users who are usually happy to help (we have no major problem recruiting for user tests). Instead of asking random people, we decided to “guerilla test” our own users. In this way, we create a virtuous cycle by giving them an opportunity to influence the product.

We wanted it to be as smooth as possible for our users so that they can do it whenever they want, with the device they want and without being too intrusive. This is why we thought about sending them an inbox message directly to their BlaBlaCar app. When they log into the app, they see the inbox message, click on it if they want to and are taken directly to a Maze survey.

This was possible thanks to the amazing work of our CRM team. Using our CRM tool, Braze, we were able to add some criteria to target the most relevant audience: only new drivers in Brazil for example, or only heavy-user passengers in Russia, France and Ukraine.

To boost the number of responses, we can also send a push notification to users.

The setup is not time consuming for the CRM team (1–2h) and we usually send batches of messages to 1–10K users, monitor the number of responses we have within one day and stop the sending when we reach the number of responses we need.

Here it what it looks like from a user’s point of view:

Step 2: Finding the right tool for the guerilla test

We decided to use Figma (our product design tool) and Maze, mostly for its user friendly and engaging interface. From a user’s point of view, there is no need to download any app to record their feedback and actions. And we observed a pretty high ”conversion rate” (more than 50%) of the people who completed the Maze survey versus the number of people who opened the inbox message.

From a research and product point-of-view, we loved the simplicity of drafting a survey. To share the results, we use the visual report Maze creates and share the main conclusions by Slack.

Step 3: Set some rules.

• Guerilla tests have to be simple: simple scenarios, non-ambiguous open-questions, no more than 5 questions and alternate open and closed ended questions.
• Guerilla tests should require low effort, for us and for the users (not more than 10 min to be completed).
• Even if it’s a low effort technique, every guerilla test project should aim at facilitating decision-making and thus should be linked to a product, design or business decision.

Users reacted positively to this new testing, especially because we kept it very simple with a few questions, but also because we crafted a dedicated tone of voice and graphic style to make it recognizable.

Tarantino’s 2nd tip: Unique, memorable style

Tarantino is well known for his memorable aesthetic of violence, with long and crude dialog and bloody scenes. In our guerilla tests, we also wanted to have a “memorable” and recognizable tone of voice and visual signature.

• Tone of voice: we kept BlaBLaCar’s fair, punchy, transparent and warm tone of voice and adapted it slightly to be a bit more familiar and engaging, adding emojis for example. Our narrative is based on asking users for their help and making them feel a part of the product design process
• Visually: we created a dedicated illustration for this type of testing so that users can recognize it immediately. We also customized Maze surveys with our corporate identity.

Tarantino’s 3rd tip: Wide-angle shots

As we saw that users responded super well to this type of testing, we decided that, whenever the project requires it, we would target a larger audience with more than 100 responses in 24–48h. We still ask open-ended questions and alternate with close-ended questions.

Example of results for a guerilla test in Russia with +170 responses

What’s next?

We tested this remote guerilla technique with 2 pilot projects and it proved its value with low effort and cost! If we go back on the KPI listed at the beginning of the article, we can be satisfied:

• We reach more users than expected: approx 15 users in less than a day. If needed we can also reach more than 100 users within 48h
• Feedback is valuable enough to influence decision-making and testing content in the product is the “first use case”
• We collect responses within 24–48h

👉 Our next step is to empower the whole Product team, so that any product team member can use the guerilla testing method and tools on their own, bringing the user’s voice to every moment it’s needed.

Does this make you want to (re)watch a Tarantino movie now?

Many thanks to anyone who participated in making it possible 👏, and especially to Ardena Gonzalez Flahavin, Akshita Agarwal, Manuela Yamada, Guendalina Esmeralda Braga, Marion Damiens, Marion Heroult and Aleksei Sushkov.