Wikia’s Guide to User Testing
Collecting Insightful Feedback With Little Time and Money
In a digital age where our attention spans have dropped to eight seconds it’s more important than ever to present users with intuitive products that add value to their lives. Gathering ongoing user feedback is a key part of building these products.
So how can you possibly collect insightful user feedback with little time and money? At Wikia we have ironed out a process that allows us to do just that.
Overview
Setting up and running in-person user tests can be very time-consuming, so it’s helpful to break up tasks among team members if possible. We break down our process into five steps:
- Test Planning: determining how the test will look
- Finding (Reliable) Participants: rounding up users to test
- Prep Work: setting up the testing environment
- Running the Sessions: taking users through the test
- The Debrief: organizing and sharing results
Test Planning
To get things started you need to create a gameplan. This step can be broken up into four tasks:
- Determining what to test
- Choosing the test audience
- Creating the prototype
- Writing the script
When choosing the test audience you may want to consider testing across multiple user segments to get a better idea of how the new feature will perform with your entire userbase. To clearly understand the test results ask a few questions about their usage habits in order to provide context as to why the answers may differ.
Although you’ll have time to fine-tune the prototype you need a working version to reference while writing the script. The success of the test is closely tied to how well the script is written in coordination with the prototype; the last thing you want is a participant getting lost or confused because the two don’t match.
Finding (Reliable) Participants
Finding and scheduling quality testers can be time-consuming. During the recruitment phase you’ll want to cast a wide net and focus on people who respond in a timely manner and have access to reliable transportation.
Here are a few ways to gather a list of potential testers:
- Craigslist: A great platform that offers free ad space and allows you to communicate with applicants via email.
- Google Forms + Craigslist: This is good for filtering applicants to better match your user segments. Note that applicants should only respond to the ad using the form; this will help weed out flaky testers.
- Existing User Base: Perhaps the best route to go if you have it, running social media campaigns and/or site-wide ads allows you to pool interested testers directly from your user segments.
To help with the process it’s good to offer compensation as an incentive in return for their time. Pre-loaded debit cards or Amazon gift cards should do the trick.
After collecting the list of applicants try to schedule them over the course of 2–3 days; the more testers you can schedule per day the better. This will cut down on setup time and help streamline the testing.
Prep Work
Now that you have your prototype, script, and participants ready it’s time to setup the room. First you’ll need a way of capturing video during the tests. We use this camera that floats above the participant’s hands and connects to a computer. This allows us to capture his/her hand movements along with what’s happening on screen.
Another great way to capture video is with a screen recorder on the testing device. Free apps such as Lookback (iOS) and AZ Screen Recording (Android) are great for this.
When setting up the room you’ll want to position a chair off to the side or opposite of the participant where the moderator can sit. Make sure you’re facing the participant and that your computer screen is facing away from them. To provide team members access to the sessions open a video conference line for them to follow along. You can do this directly on your computer or on a separate device, just make sure they are on mute.
When running the session it’s important the each participant feels comfortable. Keep the room as empty as possible in order to avoid distractions and make sure your setup gives testers breathing room so they don’t feel boxed in.
Running the Sessions
You’ve put in the work and now it’s time to execute. To ensure the sessions run smoothly be sure to run through the test a few times beforehand. Here are the key things to focus on during your session:
Introduction & Icebreakers
Getting off on the right foot is key to a successful session. You don’t have all day with each user, so the sooner they feels comfortable the sooner you start getting valuable feedback. It’s good to be casual and conversational to start. Ask them how their day is going, if they did anything fun over the weekend, what TV shows they’ve been excited about lately, etc. From there it will be a smooth transition into the disclaimer.
The Disclaimer
This section is not overly exciting but still very important, so it’s perfectly fine to read it off the script. Your disclaimer should set the expectations for what each participant is about to take part in, that it will be recorded and only shared internally, and that any feedback — negative or positive — will be very helpful. You want to create an environment that helps them feel they can speak openly and honestly. Before beginning the test be sure to get their permission to record the session and answer any questions they may have at this point.
Facilitating the Test + Tips for Success
Once the session has begun it’s important to keep the participant on track. Let them know that any questions won’t be answered until after the test since you want to see how they would react without somebody there to help. In the event they still ask one during the test respond with “Do you see a way to complete this task? How could you imagine this working?”
One of the advantages of conducting in-person user tests is being able to go off script and dig a bit deeper. You don’t want to do this all the time, but when appropriate feel free to ask questions like “Can you expand a bit more on that?” or “What do you mean by ______?”
As you begin to run through more tests these techniques will become easier and more natural. It’s important to practice on team members beforehand so you feel comfortable running the test with actual users and to make sure the prototype and script work well together.
The more relaxed and fluid you are the more comfortable testers will be, and instead of feeling like they are filling out tax forms it will seem like a casual conversation which will lead to more honest and valuable feedback.
Wrapping Up + Answering Questions
After finishing with the test have them navigate back to the first screen of the test and give an overview of how they think it went. This is a great time for additional feedback and to answer any questions that may have come up during the test. It’s also good to walk through the decision process on any questions they may have to help bring them into the design process and show their time and feedback is essential to the success of the product.
The Debrief
After you’re finished with the sessions it’s time to collect feedback and take action. Hopefully you’ve had someone taking notes during each session to help speed this part up, but if not the recordings are a great way to go back and capture highlights for any examples or statistics to help inform product decisions. I find it helpful to write down direct quotes and behaviors of each question or task so I can easily match them with other tests and create reports, such as 8/10 users had a hard time finding the “View All” button.
After the feedback is documented its time to pull out common themes and place them in a Keynote, PowerPoint presentation, or something as simple as a Google Doc. When creating the deck provide background for what was tested, what you were looking to gain feedback on, and the number of users tested. This gives context for what the results mean and how they will help in future product decisions. Wrap up with a slide for “Next Steps” and how the results will be utilized.
Now that you have done all the hard work it’s time to show off the goods. Set up meetings with the appropriate teams across the company and be sure to highlight how essential user testing can be to the company’s success.
Testing Often
When you’re finished with your first round of user tests make sure to look back and reflect on any potential changes to the process to help it run smoother next time. To help stay on top of things set up a central repository with the following:
- A set of easy-to-use templates for planning and documenting results
- Generic testing script to use as a starting point on new tests
- Centralized “User Testing” folder that includes subfolders for each product or feature tested
- Artifacts from previous tests such as prototype screens and links, the script used, and results from each tester (video + notes)
All of this helps testing become easier and more efficient while also serving as a great way to look back on past findings. Testing early and often can help take your company to the next level, so it’s important to build a testing culture within your team.
And that does it! We covered planning for the tests, how to find (reliable) participants, setting up the test environment, running the sessions, and sharing results with team members. With this process in place user testing can be a core component of your product’s development, even if you have little time and money. Now get out there and start testing!
