User tests for bootstrapped products
Hi. I’m Clo. In October of 2018, I started working on my first solo project: a relaxing mobile game called Bloom.
There are 4 chapters in Bloom. I wanted to make sure that they were neither too easy (where’s the fun if there’s no challenge?) nor overly complicated (which may lead to dissatisfaction and giving up the game). Observing users interact with Bloom could help me do that.
This isn’t meant as a strict guide to follow to the letter. Whether it’s time, money or people, we all have limited resources, even more so when it comes to bootstrapping. This is a rather exhaustive article, but you should absolutely feel free to adapt it to your own product and constraints.
What’s the point of user testing?
Gathering feedback
Whatever you’re working on, don’t rely on your assumptions alone to judge the result. Ten people’s experiences are more reliable than one person’s only. Additionally, it’s your work so you‘re in too deep to have hindsight. Listen to others, watch them use your product, observe how they react and what they understand. Take note of what’s hindering them and what’s moving them.
Pro tip: this is not only for tech-related work, but it’s also for anything that’s designed. You can ask your friends to check your article before it’s published. You could also ask people to check your code, to be sure it’s understandable for the next person working with it.
Using that feedback to improve your product
The insight you gather will allow you to improve your design. The more people you interview, the more qualitative feedback you will get. Besides, the frequency of each remark will help you weight their importance. This will be convenient for deciding which points need more work. Because yes, there’s a chance you don’t have time to work on everything — you may have other priorities.
How to prepare user tests?
I started by testing the first level of the game. To make the most of it, I asked questions that had to do with both this specific level and the whole game.
Research objectives
Your research objectives are broad points you want to know about.
Text looking like this means that I’m illustrating my words with the case of Bloom.
My research objectives included:
- The game’s usability
- Users’ understanding of the level’s scenario
- The level’s level (see what I did there?) of complexity
Methodology
Among other things, research can be qualitative or quantitative.
- Qualitative: studying human behaviour via observation and context (focus groups, guerrilla testing, etc.).
- Quantitative: studying human behaviour via numerical data and statistics (surveys, eye-tracking, etc.).
For user tests, I obviously went with a qualitative research (my favourite type). In my opinion, it is by far the most relevant way to go for bootstrapped products. As the maker, you need to guide users through discovering the product. You also need to be there in real-time for any question they may have. You want to see and hear their reactions.
I also chose to conduct semi-structured interviews (as opposed to structured and unstructured ones). This means that I prepared a set of questions I wanted to explore, but we could sidetrack from it. In semi-structured interviews, questions are quite open and participants can elaborate more on each topic. It’s an explorative approach. Interviewees were free to bring up elements that I didn’t mention. Returning to the pre-determined question was up to me.
Finally, since my target users are everywhere on the planet, I went for a majority of remote user tests. You could also choose to have them all face-to-face if that makes sense for your product.
Research questions
It’s time to go into detail about your research objectives. These are specific questions you want to know about.
Some of the research questions for Bloom were:
- [usability] Are the d-pad buttons big enough? Are users sometimes pressing next to them? Are users OK with the d-pad being on the bottom-right corner?
- [understanding] Do users understand that the green dot is the turtle? Did users expect something else as the representation of the turtle? Are users OK with the turtle not being animated?
User journey
This is the flow your user will follow while going through the interview. If you decide on a very specific user journey, you should be aware that users might also take an adjacent road.
Since I was testing level 1 as well as some aspects of the game in itself, I divided the journey into 3 broad steps:
- Before playing: what users can already understand from the starting screen.
- Playing the level: what they observe and feel during the game. Is anything different from what they said during step 1?
- After playing: their thoughts in hindsight. They can take the time to revisit the level.
Participants
Criteria for recruiting participants can be more or less strict and extensive. The important thing is to make sure that they match your target users. Criteria could be about age, profession, interests, habits, etc.
Look for participants in the same place you’d look for customers. Your list of criteria will guide your search. You can also use an incentive to have more people register (e.g., offer financial compensation).
When recruiting participants, mention how long the test may last. Send them an electronic invitation with all the details, so that they have everything in the same place. It’s also a matter of UX: make sure your invite (and the whole test) is a good experience for the participants.
Finally, keep in mind that some participants may cancel. Be sure you have enough volunteers so that your study will still be representative if some are unable to make it.
Bloom’s target is people interested in:
- mindfulness
- indie games
- relaxing experiences
- … and who have an Android phone
I emailed the kind humans who subscribed to Bloom updates. I also asked friends and Internet makers if they would like to take part in my research. For future tests, I could reach out to people on websites and forums about indie gaming and mindfulness. It’s also possible to contact game designers and developpers. I kept in mind to have balanced genders and balanced tech-savviness.
In my invitation, I explained the steps to install TeamViewer (a screen sharing tool I used for the tests) and provided the direct link to the app.
I also shared my Skype and/or Hangouts username. In case we hadn’t decided on what service to use, interviewees could pick their favourite one.
Detailed workflow
It’s now time to combine your research questions with the user journey you decided on. You can detail the workflow screen by screen or step by step. This allows you to match each research questions with a screen/step and a component on that screen/step.
I didn’t create a detailed workflow because I’m working on Bloom alone, and I have time constraints. Here’s an example of what I could have done.
Question: ‘Do users understand that the green dot is the turtle?’
Step: Step 1 — Before playing
Component: The green dot
Testing protocol or testing guide
The testing protocol is a document gathering everything you need to conduct the tests. It’s composed of the following:
- participants criteria
- research objectives
- participant briefing
- detailed workflow
- participant debriefing
I had an abbreviated testing protocol with me during the tests.
Tools I used
All these services have free plans.
To conduct remote user tests, I had interviewees download TeamViewer QuickSupport. For my part, I had downloaded TeamViewer for Remote Control. While the remote control feature was useless, it’s the only free app that allowed to share a mobile screen. Streamers only had to share their ID with me, and I could see their screen.
I used Notion to build my testing protocol. I used a table to write down and visualise the feedback I got as soon as I received it.
I used either Skype or Hangouts to call participants, depending on their preference. Any international calling service would work.
A calendar is useful for sending invitations to participants. It’s a good place to include the interview details. It also reduces the chances of someone forgetting about the appointment. Calendly is also a great way to schedule interviews. You can specify your availability in time slots, and send your link to participants. Then, they are able to pick whichever slot works best for them.
How to conduct user tests?
Introduction & briefing
Introduce yourself and ask a bit about the participant (their identity, their passions, their hobbies, etc.).
Make sure to mention that you’re gathering information for research purposes only. It’s important to state that you won’t share it with any 3rd party (especially if you are recording).
Explain what the product is about.
Explain that there’s no right or wrong answer, that it’s the product you’re evaluating. Tell the participant you’d like them to think out loud during this whole test. Be comfortable and make sure the tester is comfortable as well.
Finally, before you begin, ask the tester if they have any question.
Here’s a way to introduce what the participant is about to test:
‘Bloom is a relaxing mobile game about a turtle growing up. You are about to play the first level. It focuses on breaking the shell the turtle is in.’
The test in itself
Follow the questions outlined in the testing protocol, to some extent.
Since this is a semi-conducted interview, let the user speak for themselves, and listen. Don’t suggest answers to your own questions. Are they stuck? Good, now let them think and see if they can figure it out on their own, just like in real-life conditions.
If they really need your help to progress, assist them in the most neutral way possible. The less you influence their remarks and behaviour, the more realistic your results will be.
A few neutral questions I asked:
- ‘What do you think will happen if you do that?’
- ‘How does that make you feel?’
- ‘What do you think you should do next?’
- ‘What do you understand from this?’
If you don’t record (which I didn’t), take a lot of notes.
Write both insights and quotes. If you’re working for someone else, the quotes will help you present the insights.
Insight: She didn’t understand the green dot was the turtle.
Quote: ‘I’m not sure what it is, but I guess I’ll find out when I play.’
Debriefing
Thank participants and explain how their understanding of the game will help you. Ask if they have questions and if there’s something else they want to share. I’ve noticed that participants who have more to say are likely to keep playing while they speak.
Since this is an iterative process, you can ask if interviewees would like to take part in a follow-up test. If that’s the case, let them know when you think it could take place.
If you want to share publicly who joined in your tests, do so with their permission only. It’s a matter of privacy.
I always asked the participants before tagging their Twitter handle in a thank-you tweet.
I also reminded them that if anything else came to mind later, they could always message me.
Thank you for reading!
Need guidance with UX Research? I can help with that! Feel free to contact me through my website or on Twitter, and I’ll be in touch shortly.
If you’re curious about Bloom, you can subscribe here to get updates (should I mention that I don’t spam? I don’t spam). You can also follow me on Twitter, where I share my progress regularly.
I would like to thank the following people
- Isabel Costa for her outstanding proofreading
- Mohammad Rajabifard for his additional ideas
- Marie Denis for her friendship and for the amazing WomenMake community
