How to Conduct a User Study on a Budget
If you’re working on a new product idea or enhancement determining if your users will understand and identify with what you’re planning is key.
It may be tempting to skip the user study and save your money for feature development and marketing; but you risk building something that people won’t use or understand.
Professional usability firms use many methods both qualitative (aiming at collecting insights) and quantitative (aiming at statistics) to derive information. Quantitative studies require statistically significant numbers (at least 20 users). This is very expensive because qualified test users are hard to come by and require systematic recruiting to represent your target audience. Luckily, quantitative studies are not required to improve usability. “You don’t have to measure usability to improve it” — Jakob Nielsen, Nielsen Norman Group.
You don’t have to measure usability to improve it.
Usually, it’s enough to test with a handful of users and revise the design based on a qualitative analysis of their behavior. When you see more than one person confused by the same design element, you know you have a problem. You really don’t need to know how much users are being delayed. If it is confusing to users, you need to change it or get rid of it. You can conduct a qualitative study with 5 users. That makes it 25% of the cost of a quantitative study. Furthermore, qualitative studies can and should be conducted early and often. “Doesn’t matter whether you test websites, intranets, PC applications, or mobile apps. With 5 users, you almost always get close to user testing’s maximum benefit-cost ratio.” — Jakob Nielsen, Nielsen Norman Group.
Doesn’t matter whether you test websites, intranets, PC applications, or mobile apps. With 5 users, you almost always get close to user testing’s maximum benefit-cost ratio.
At Flower Press Creative Studio we are often approached by clients who have a new product idea or enhancement request. We advocate for early testing, before development dollars are spent. Here is a basic approach you can take to improve your product design or feature enhancement concept:
- Sketch out a few screens using a pencil, on paper. If you’re versed in design tools you can create simple black and white wireframes using Illustrator, Sketch or an online prototyping tool.
- Take photos of the screens, or export PNGs and link them together using a prototyping software like InVision
- Decide who you want to talk with. Set up a recruiting form using a free Google Form and ask questions to validate your study participants.
- Share the link on social media. Be sure to offer an incentive like coffee, pizza or an Amazon gift card for participating. This shows respect for your participant’s time and will help get more traffic to your recruiting form.
- Come up with a set of features you want to receive feedback on. Perhaps it is the introductory page and registration flow.
- Write up a short list of tasks you would like users to complete. Be sure not to use language contained within the user interface within your instructions. This can lead to false results.
- Set up a time to meet. Coffee shops with a quiet corner, libraries or co-working spaces are perfect for this. Test the wifi connection ahead of time.
- Load up your prototype. Ask permission to record your conversation using your smartphone. This will keep you free to observe them.
- Give your participant a quick introduction and set of instructions. Ask him or her to “think aloud” so you have some idea of what is going on in your participant’s head. Watch his or her behavior, facial expressions, and pauses. If your participant stops for too long, ask what him or her what he or she is thinking. Try to be open and non-judgmental. People tend to blame themselves for not understanding how to use things. Make sure to tell your participant the prototype is early, it’s not their fault, you’re working out the kinks.
- When the task is completed ask for overall impressions of the system, suggestions and feedback.
- Thank your participant.
Once you’re back at the office, review your recording and write up notes to share with your team. Do not wait too long, as impressions from your interview can be quickly forgotten.
Once you meet with a few study participants, you will see themes emerge. For example, a particular menu or button placement may be confusing. A particular feature may be more exciting than you expected. And a feature you are excited about may not resonate with users.
We are often surprised during our user studies with how participants react. Sometimes features are well received. Often small “side features” end up becoming the most exciting part of the experience. Testing early, and often allows us to inform our clients about where to focus their time and energy. It allows us to save development dollars building wasted or unclear features.
As you iterate on your design, setup additional studies to validate your solutions. Over time you will come away with a successful product that you can improve and enhance as your user-base grows.
Engaging in user studies can be very helpful for shaping and refining your idea. Don’t leave it to your interaction designer. Get involved. Entrepreneurs, studio owners, visual designers…everyone should be talking to users directly.
Khaiersta Flowers English is Founder and Creative Director of Flower Press Creative Studio, a digital product design and development studio in Seattle, WA.
