How to do user testing on a budget
A beginners guide to run your first user testing session for $0
User testing is expensive. Designers, researchers, product managers, or developers are often tasked with user testing on limited budgets. As a designer, I’ve worked on numerous projects under similar constraints and I’d like to share some tips and tricks I’ve learned throughout the years.
Though I’ll focus on testing methods for digital products, the general concepts can apply towards services or physical products. The goal is to show ideas to users early to evaluate, iterate, and validate assumptions.
What is user testing?
User testing, also known as usability testing, is a process that identifies product issues by testing it with real users. User testing has roots dating back to World War II where testing was used to improve military equipment.
Steve Krug’s Don’t Make Me Think further popularized user testing across the industry. Nowadays usability methods have been made available beyond the traditional confines of military, academia, and corporate R&D labs. If you’re interested in the history of user testing, check out Jeff Sauro’s usability timeline.
Why do you need to do user testing?
Its a lot cheaper to catch potential problems early within the product development life cycle. Tech companies typically follow an agile workflow that looks something like this:
This graph represents how projects are passed through teams across the company. Product Managers start by defining product requirements that are passed along to the UX team. After finalizing design, the product moves through Development, QA, UAT, and Release Engineering. This cycle repeats itself in an iterative manner until the final product is complete.
Designers who rely on intuition and assumptions alone must wait until the release of each cycle to gauge feedback, potentially wasting a lot of resources. Reversing the damages of poor user experience is an expensive and arduous process. Once layout and patterns are established, designers must thoroughly capture evidence of poor usability and convince others to implement new patterns.
“If you think good design is expensive, you should look at the cost of bad design” — Ralf Speth
User testing was developed as a quick and inexpensive solution to mitigate some of these risks. You can validate assumptions, steer product and design direction more confidently, and uncover hidden issues using actual user feedback. With that said, user testing has limitations due to the artificial testing environment and small sample of users. It will never substitute feedback from released products where mass users are engaged with real data.
Get started with free tools
The simplest and most inexpensive method, paper prototyping, allows non-designers to be involved in creating a simple interface. I prefer using wireframing and prototyping apps built specifically for digital products. We’ll start by using apps that are free or have free trials:
- Wireframe — Sketch, Balsamiq, or Draw.io
- Prototype — Invision
- Screen sharing — Google Hangouts, Skype, or Validately
The screen sharing tools are for remote user testing, which means testing through a video conference call.
Step 1 — Identify reasons for testing
Its important to first identify what questions you are trying to answer through user testing. Start by looking for potential pain points, complex user flows, and other ideas needing validation for your product.
With questions in mind, researchers typically start by choosing the best testing method and creating a usability test plan. Since we lack resources, we’ll stick with one of my favorite testing methods called thinking aloud.
“In a thinking aloud test, you ask test participants to use the system while continuously thinking out loud — that is, simply verbalizing their thoughts as they move through the user interface” — Jakob Nielsen
Step 2 — Create a script
A script is a printout document for moderating a user testing session. Your script should contain an introduction stating research goals and house rules, some probing questions, and then jump into performing key tasks. For example,
“Hi Fiona! My name is Julie and I will be moderating this session today. Thank you for taking the time to participate in this session, your feedback is very valuable. Before we begin, I would like to go over a few items with you…
Lets pretend that I’m running a testing session for an app called DogeMaps. After identifying research goals in Step 1, I will brainstorm some questions around these themes:
- Background/Context— “How do you usually get around the city? Do you use any apps on your phone?”
- Expectations — “Before clicking on the ‘search’ button, what do you expect to happen?”
- Usability — “How do you save this location for future reference?”
- Rating Scale — “How difficult (1) or easy (5) was it to search for San Francisco on this map?”
Heres a draft script to help spark some ideas. If your product is complex, prioritize your questions and conduct multiple rounds of testing. For a 20–30 minute session, aim for 1–2 intro questions and 3–5 tasks.
Tip: Pay attention to wording and avoid leading questions that influence user behavior. For example, “How do you scroll to see more results?” suggests that I can scroll. A better question might be “Are there more results on this page? If so, how do you see these results?”
Step 3— Create wireframes and a prototype
Wireframes should focus less on visuals and more on layout, user flow, and copy. Keep your wireframes simple to save time iterating through prototypes. The general rule of thumb is to create high-fidelity mockups after validating core user flows.
The next step is to create an interactive prototype using Invision. Check out Invision’s QuickStart Guide to learn how to link wireframes together. The prototype must follow the same order of questions outlined in the testing script. If task #1 is “how do you login to DogeMaps?” then the prototype’s first screen should be the login and the second screen should be the dashboard.
If task #2 is “how do you change your password?” then the dashboard needs to link to the password reset page. Imagine users trying to complete task #2 by clicking everything on the dashboard. Its not necessary to link every button on each page; simply take notice and redirect them towards the right path.
When your prototype is complete, grab the private URL from the first screen. In Invision under ‘link settings,’ make sure ‘Hotspots enabled’ is turned on and ‘Prevent hotspot hinting’ is also turned on. Copy this URL and store it for future reference.
Step 4— Recruit users
LinkedIn’s audience is GenX and GenY professionals while Snapchat’s audience is GenZ teenagers. Brainstorm the target audience for your product and provide incentives if you have a budget. Larger companies like Google or Airbnb typically incentivize users by offering a generous gift card. If you don’t have a budget, here are a few tips for recruiting.
If you already have users…
Identify people who are actively engaged with your product and send a short message/email. Offer a sneak peak into the latest features and the opportunity to shape your product’s direction. Engaged users are more likely to give feedback if they have a vested interest in your product; your success equals their success. If users aren’t engaged, explain how your product brings value or solves their problem.
If you have no users…
Think about places where your target audience congregates either in person or online. Reach out to special interest clubs, friends of friends, Meetup, Reddit, online forums, etc. Heres a sample message for recruiting:
Hi Billy! I’m the founder of DogeMaps and I noticed that you were an avid user of GoogleMaps. I would love to get your feedback on DogeMaps — its an app that saves you 50% more time commuting than GoogleMaps. Would you have 20 minutes to hop on a google hangouts call? If so, what are the best available times for you?
If all else fails either broaden your user base, ask friends and family, or provide small incentives. The logistics around recruiting users can be quite time consuming from my experience. If you have a larger budget and plan to recruit many users, companies like usertesting.com or ethnio can make your life easier.
How many users should I recruit?
3–5 users per round is enough to catch most problems. Spread your resources across different rounds of testing if you manage to recruit more users. Lastly, you may want to tread carefully through legal waters and have users sign a consent form.
Step 5— Setup your testing environment
Its always preferable to conduct in-person testing sessions at quiet office spaces. Cafes are also popular but can be distracting with all the background noise. Wherever you choose, make sure there aren’t too many distractions.
Remote sessions are easier to recruit for and can be conducted in the convenience of your own home. However, be prepared for technical glitches and always have a backup plan. I’ve experienced users unable to connect screen sharing, mics not working, and prototypes malfunctioning.
Always conduct a trial run!
I’ve discovered prototypes that were incorrectly linked during actual user testing sessions. Don’t make the same mistake and always conduct trial runs to make final script and prototype revisions.
Step 6— Conduct the session
I usually send a reminder email the night before a remote testing session. It includes the prototype URL and instructions on how to setup Hangouts/Skype/Validately. The faster you get through logistics beforehand, the more time you have for asking questions.
Reminder emails are also useful to reduce no-shows, and I’ll typically wait 15 minutes before calling off a session. If its your first time moderating, you might feel nervous or unsure of what to expect. Don’t worry, you’ll progressively develop your moderating style and learn how to ask insightful questions.
A great way to develop user empathy across teams is by encouraging people outside of UX (Project Managers, Developers, QA, etc) to observe these sessions. Keep at least 15 minutes between sessions for users who run over their allocated time.
Find a Notetaker
Its impossible to take good notes and moderate at the same time. An ideal session should have at least 3 participants — 1 user, 1 moderator, and 1 notetaker. The notetaker’s job is to record both what the user says and does, especially behavior that deviates from said statements. For example, a user might say “I think the search button is very intuitive” after clicking 7 different areas before finding the search button. Be cautious of what users say and focus more on what they do.
If you can’t find a notetaker, use recording programs like Quicktime or Validately to review your sessions later. Be sure to disclose this both in your recruiting email and on your consent form.
Step 7— Distill key findings
The final step is to distill and share key findings using the Feedback Capture Grid. Draw a grid with the 4 quadrants and symbols above. The top left quadrant represents ‘user likes,’ top right represent ‘constructive criticism,’ bottom left represents ‘questions,’ and bottom right represents ‘ideas’. Transfer your notes onto post-its and place them into different quadrants, grouping together common themes of post-its. The end goal is to identify patterns and spark ideas within each category.
Revise, test, repeat
User testing will leave you with actionable next steps for your product. Products should start development when usability issues are addressed and general feedback is positive. If feedback is neutral/negative or the value proposition isn’t clear to the user, revise your prototype and run another round of testing.