Conducting guerrilla tests in preparation for OHIP+.

12 lessons learned about guerrilla testing

When it comes to designing digital services, getting feedback from users right away is important to make sure we’re headed in the right direction.

As Elizabeth mentioned in her earlier post, we’ve been using guerrilla testing — testing rough prototypes with a variety of people, randomly selected, in a very short amount of time — to get a clearer sense of the direction of the new OHIP+ site.

Guerrilla testing is an excellent way to get feedback on how to consistently improve and build upon our services. It’s not the only kind of testing we do, but it’s an important part of our toolkit: it is fast, cheap, easy to do, and doesn’t require as much time as other kinds of testing.

We know that there are few things scarier than putting your hard work out in the public and facing the critique of strangers. We also know that approaching strangers and getting rejected can be super daunting. That said, we think this kind of testing is amazingly useful, and wanted to share a few things we’ve learned about with you:

1. Location is a factor in the willingness to participate. We recently did a guerrilla testing session at the mall and noticed that a lot of people were reticent to participate. This could have been because, when in a mall, a lot of shoppers are approached by advertisers and salespeople, and already are hesitant to engage with strangers.Being aware of the environment, and the user context to that environment, is an important consideration.
2. Be confident when approaching potential testers. Don’t hesitate, just go for it. Too much hesitating and private back-and-forth conversations with your colleagues add an air of secrecy and awkwardness — exactly what you want to avoid.
3. Approach potential participants at eye level. Speaking to someone while they sit and you stand creates a subtle power dynamic that doesn’t always create the best testing environment.
4. Be casual. When you’re testing with real users, out in their natural environment, you don’t need to be formal. If you are casual, you feel like more approachable. Mirror the formality and dress as the people you are trying to approach. It will make you feel like part of the group, and will make people more likely to speak to you.
5. Be aware of how many people are conducting the test. Having a second person with you is incredibly important and useful: they help take notes, provide support during the testing session, and help you feel more approachable. However, approaching testers as a large group could be detrimental — a big group can intimidate the user.
6. Master your elevator pitch. Here’s a sample line we use when approaching people: “Hi, I am here with my colleague and we are part of Ontario Digital Service. We have a website that we’re working on, and would like to hear your feedback. I was wondering if you might have 10–15 minutes to go through the website with us, and in return, we’ll give you a gift card for your time.” Give them time to think, and wait for their response. Keep it short and simple.
7. Be prepared for rejection. It will happen — sometimes people just don’t want to participate.That’s okay. Get used to saying thank you and walking away. There’s always another person you can approach.
8. Introduce yourself and your roles. Once someone has agreed to participate, be very clear as to who you are, why you’re there, and what each of you will do during the session. After the individual has consented to be interviewed, start with:: “My job here is to guide you, while my colleague takes notes on improvements we should make. They will make sure that we don’t miss anything you have to say.”
9. Keep checking for comfort and ease. During testing, monitor the visual and verbal cues of the participants. If you feel that someone is uncomfortable, acknowledge their feelings and try to understand why. Ask them if there’s anything wrong or if they need further clarification. If they continue to feel uncomfortable, ask them if they want to stop the test. It’s better to get good feedback from someone who is happy to share, than having someone feel forced to share their point of view.
10. Make sure your test makes sense to the user. When designing your tests, imagine that you are a person that has never heard of the service. Never set up tests that you know will fail (even if it’s for testing purposes), because this discourages users and they start to blame themselves. Try to understand why they are interacting with your products a certain way.
11. Avoid leading questions. An example of a lead-in question is: “would you click on this button?” With this question, you are already setting them up to do something that they may, or may not, have noticed on their own. Instead, ask “how would you access this program?” It may be obvious to you that the user will click on that specific button, but you may be surprised by their actions. Follow what users are doing, rather than lead them down a path.
12. Have backups. Always. It doesn’t matter if you’re testing a live website, screenshots, or paper prototypes. Always have something handy as a backup because software, hardware, and product malfunctions do happen.

Happy guerrilla testing, folks!

Conducting guerrilla tests in preparation for OHIP+.

Lucia Hsieh is a User Research Lead in the Ontario Digital Service. She has lived in 6 countries and speak 3 languages. Lucia strives for a minimalist and sustainable lifestyle and enjoys good food and company.