UX Design: How to set up a user testing session

Published in

NYC Design

6 min readFeb 5, 2020

A short introduction

A very broad field, “UX is an umbrella term for any activity or profession that contributes to a better experience for the user.” (Krug, 2002) This means UX’s focus is to guarantee that users have the best possible experience while using a product or service.

This is done by collecting what we call “evidence” — opportunities to get feedback from people in your target audience — and it is a great way to reduce risks and ensure constant improvement. UX is an approach that (in optimal conditions) should already be considered when the fundamental concepts are being developed — ideally before the interfaces are designed.

Why does UX need to come so early in the process?

If we designed first and researched after, we would have to incorporate changes into our designs to meet the needs of the users that we want to speak to (our target audience). The goal of user research is to discover how a user might experience a product and adapt the design to address it. It is a way of avoiding mistakes, or situations that are less than optimal.

Many clients wonder “That sounds complicated. Do we really need to research our users?” Well, the answer is: yes! Without user research, we have no evidence, which means our work can only be based upon our own experiences and assumptions. We like to think that all users behave just like us, but that is not true. If we want to develop a good product, we need to gather feedback from our target audience.

We like to think that all users behave just like us, but that is not true. If we want to develop a good product, we need to gather feedback from our target audience.

User research can reveal key findings to guide design choices and improve the overall experience of users. Even the best designers need evidence because UX is about meeting your audience’s needs. Without evidence, UX is just UI. (If you don’t know the difference between these two, read this article.)

So how do we collect evidence?

There is more than one way, depending on if you want to make a completely new design or an update. For this article, let’s assume that a website already exists and you want to make an update.

For starters, if you aren’t a designer, start by involving your designer in the process, or hiring a UX designer. Using a tracking tool, this professional can analyze your statistics, interpret heat-maps, see where A/B testing might be helpful, and test the existing interface in other ways.
But if you want to really find usability issues and reach a deeper understanding of your users, performing user tests is the way to go. It is by far the best way to connect directly with your users and see how they use your product/service.

If you want to really find usability issues and reach a deeper understanding of your users, performing user tests is the way to go.
It is by far the best way to connect directly with your users and see how they use your product/service.

User testing 101

Although the principle is simple, lots of work goes into organizing a testing session. We will try to break down some basic aspects of how to begin:

Define your testing goals. What are you trying to learn?
Define how many users will be needed and what kind of profile they should have. Tip: the more diverse, the better!
Write task scenarios that support the user goals you designed for.

How to organize a testing session

You will need at least three people per session to get a testing session going.
Here are the three main roles involved in the process and what each of them does:

Also very important is the setting of your room. We could get very detailed, but for now, here is a very short list of how to prepare your room and basic settings:

Defining the tasks

This is a vital part of the process. You can’t just call users to test your product without having defined tasks.
“Open this page, browse around and tell me what you think” isn’t a task!
Here is some advice:

You might need to start much earlier on the process than you think. A good first task is, for example, “find this online” (on search engines). Don’t underestimate how much learning you can already get here.
If you are testing an established product, looking at the statistics could be very useful for this step. In the statistics, you can see which pages are the most visited. Therefore, finding specific information inside one of these pages might be a good task, for example.
Including an ASQ (After-scenario questionnaire) at the end of a set of tasks is almost always a good idea, since insights might come from it.
After the normal tasks, you can do a quick interview. This is a moment where questions that spontaneously pop up might be included.

Here are some ideas of things that can be tested:

Can the page generally be found easily? (SEO)
Landing Page: are the main elements and Call-to-actions easily found? Are users scrolling until the end of the page?
Navigation: how do the users navigate through the website? Is it easy? Are there navigation issues?
Can specific information be found easily? How does the user manage to find it?
Is some information the user is searching for, in front of their eyes but is missed out?
Are tools (download, shopping cart, …) placed somewhere where they can instinctively be found?
Interactive elements: are there clickable areas that aren’t being clicked? Or: Are there non-clickable things where users try to click?
Are there important elements that should be clicked, but aren’t they?
Are there pages with a big number of visits that are simply hard to find?

Criticism in disguise

After testing some of our sites with users, we always end up with many takeaways. But bear in mind that users often want to please the tester, since they know it’s your product which is being tested. You should be aware that most criticism will come disguised. Here is an example of a testing session:

“I am thinking some people might get a little confused there. Like: ‘is this shit working?’. But no no, I thought it was really cool!”
USER, 2017

While this user ended on a positive note, what they really had to say was that the page was confusing and made them doubt its functionality. This is pretty serious and isn’t how anyone wants their users to feel. So, even when a user is trying to soften criticism, make sure that as a tester/observer you are able to detect it.

A final thought

A misconception in the creative industry is that if we invest in UX and do enough user testings, we will come up with the-perfect-final-website-that-never-needs-to-be-changed-again. Sorry to disappoint you, but this isn’t a thing.

There is no perfect website that has all the issues covered. User behavior changes with time, so on the internet things need to constantly evolve to address these changes. This means that if we analyze any user interface — a website, an app, or the signalization of the underground — we are very likely to find room for improvement.