The UX research methods playbook (Part 2)
A practical guide to 20 different user research tactics.
This is Part 2 of the playbook. You can read Part 1 here.
Introduction
Welcome to the second (and final) instalment of the UX research methods playbook. Hopefully splitting it makes the content less cumbersome and easier to use.
Part 1 of the playbook covered the first ten UX research methods from A/B testing to the Five-second test. Here’s Part 2, with the final ten methods covering:
- Focus group
- Interview
- Multivariate test
- Participatory design
- Prototype testing
- Survey
- Task analysis
- Tree testing
- Usability testing
- Wizard of Oz
11. Focus group
What is it?
A method in which group discussion is facilitated by a researcher.
How does it work?
- Focus groups are one of the all-time classic research methods. You get a bunch of people together, start them talking about a subject of interest and record useful things they say — what could be simpler?
- Typically, focus groups for UX research consist of between 5–10 participants. This group is joined by a facilitator whose job it is to focus discussion on the right subject.
- Focus groups are an easy way to generate a lot of qualitative data from transcripts/recordings and the facilitator’s notes. They can be done online or in person, and require no special equipment or training. As a result, they’re an attractive and commonly used UX research method.
- There are a few potential downsides to focus groups. First, there’s a danger of loud voices dominating. Second, there’s a risk that the group will form a consensus view that doesn’t actually represent anyone’s individual thoughts. Third, discussion can go off at a tangent into irrelevant areas. It’s the facilitators job to try and mitigate all these possibilities.
When to use it
Focus groups are really flexible research method. They can certainly be used to understand users’ feelings on a specific product/service. But they can also be used to simply explore people’s life experiences, goals and frustrations generally — this helps build empathy and context which could inform a potential new solution to a problem/product.
Tip: Avoid the temptation to simply build things people ask for in a focus group. Remember, users aren’t typically in the position to know what would best meet their needs — that’s the designer’s job.
12. Interview
What is it?
A format where a researcher meets a participant to ask questions and learn from their responses.
How does it work?
- User interviews are another classic UX research method, and arguably one of the best for collecting useful qualitative data. Typically, interviews are conducted one-to-one and last about 30–60 minutes.
- They can be carried out virtually or in-person, but face-to-face interviews are best as it allows researchers to build a more genuine rapport, as well as observe more non-verbal communication (like body language).
- It’s important that participants are relaxed, so user interviews should be carried out in an environment they feel comfortable, e.g. their home, a cafe or library.
- The purpose of interviews are to build up an understanding of a single person as a user — who are they, what’s their experience, what are their goals and frustrations, how do they currently use relevant products and technology, etc. This builds empathy, helps to develop user personas, and ultimately can inform solutions to meet user needs.
When to use it
User interviews can be carried out anytime, but work really well at the start of a project or product life cycle. They help researchers understand real users in real depth. You should do as many interviews as practically possible as long as they feel useful, and you’re getting new insights out of each one. The only downside of interviews is that they’re typically time-consuming to arrange, carry out and analyse the data afterwards.
Tip: Ask mostly open rather than closed questions in user interviews to encourage users to give in-depth answers. E.g. ‘How do you feel about product X?’ rather than ‘Do you like product X’ which might lead to simple yes/no response.
13. Multivariate test
What is it?
An experiment testing multiple permutations of a design to see which combination works best.
How does it work?
- Multivariate tests are like the more sophisticated (and complex) version of A/B testing. Instead of just testing the effect of something simple like different call-to-action (CTA) button colours, in multivariate testing you’ll experiment with user journeys where multiple variables are modified.
- For example, take a website landing page: you might run a multivariate test where the heading, hero image, CTA button and lead text are all subject to change. Unlike the 50/50 user split in A/B testing, here you’ll have multiple experimental groups — with each cohort of users seeing a different mix of design elements.
- Remember the goal of multivariate testing is to test which combination of variable changes is most effective. So it’s not just a case of showing each group of users a different page.
- Multivariate testing can also be applied to multi-page tasks. So it’s not just a tool for finding the optimal landing page, but the overall most effective user journey.
- Multivariate testing generates quantitative data, and requires specialist software as well as a robust scientific method to do properly.
When to use it
Multivariate testing is an incredibly powerful technique. However, the downside is that you need a lot of data for a statistically significant result. That means multivariate tests should be reserved for large websites, user journeys with a high volume of users, and where there’s an understanding that it’ll take time to see meaningful results.
14. Participatory design
What is it?
A technique where users are encouraged to design a solution for themselves.
How does it work?
- Participatory design is an interactive UX research practice where users get hands-on designing a product that could meet their needs.
- These sessions can be done online or in-person, but like a lot of user research doing things face-to-face makes it easier to build rapport and empathy. It can also lead to more natural interactions and is more inclusive of those without digital skills.
- Physical materials used in participatory design sessions could include anything from pencils and paper to Lego blocks. Making the environment feel more relaxed and creative may well lead to better outcomes.
- The goal of participatory design is not to actually implement what users come up with as the final product, or even develop it into a prototype. (Remember: users are not UX professionals, and they’ll be unaware of things like design principles, heuristics and conventions, let alone what’s technically feasible.)
- Participatory design is more about discovering what’s important to users and what they expect from a product. It can help you understand how people use interfaces, what they get frustrated by, as well as their overall conceptual models of systems. All of this can be useful for UX designers to start prototyping viable solutions.
When to use it
Participatory design is best carried out in the early stages of a project. While it has ‘design’ in the name, participatory design is actually more aligned with UX research and discovery — it’s an exploratory technique to help understand the perspective of users.
It’s important to be clear on definitions of involving users in the design process. Participatory design and co-design are different things.
15. Prototype testing
What is it?
A method to find out how something needs to work by testing a mock-up of it with users.
How does it work?
- Prototyping can be used as a research method to elicit requirements for a product or system. During this process, UX designers create mock-ups of a UI for users to test and then improve the design in a new iteration, e.g. by adding a new feature that meets a user need.
- These mock-ups can be anything from simple hand-sketched wireframes to mid/high-fidelity prototypes created using design software like Figma or Sketch. Digital tools make it easy to quickly mock-up an interactive UI — and generative artificial intelligence (AI) makes it even easier.
- While it’s tempting to leap straight into polished prototypes that look and feel just like a real product, low-fidelity wireframes can help both users and designers focus on core concepts, layout and functionality (rather than getting distracted by aesthetics). Tools like Balsamiq and Wireframe.cc are designed to focus on simple wireframing.
- This idea of eliciting requirements through iterative prototyping isn’t new. For example, it was used to design classic software like Windows 95. It’s also constantly evolving, for example researchers have used augmented reality (AR) as an interactive prototyping technique.
- There are a lot of benefits to prototype testing: it’s a quick, cost-effective and non-committal way to share and test designs, and it’s been found to be an effective way to both elicit requirements and unpack tacit knowledge.
When to use it
Prototype testing is a great UX research method for projects that don’t need to maintain lots of detailed documentation related to requirements. This means it’s most suitable for agile, iterative projects where it’s acceptable for the latest prototype to be considered the requirement ‘spec’. It won’t be as appropriate for large-scale waterfall projects.
16. Survey
What is it?
A questionnaire-based research method used to collect responses from users on a specific topic.
How does it work?
- Surveys are one of the quickest, easiest and cheapest ways to gain insights from users. With relatively low effort, you can collect a high volume of data. For that reason, they’re one of the most commonly deployed tactics in the UX research playbook.
- There are plenty of online tools for creating surveys, such as Typeform, SmartSurvey or Microsoft Forms. These tools are designed to be intuitive, making even complex forms with logic branching accessible to pretty much everyone.
- However, the simplicity of surveys is deceptive — they’re harder than they look. It’s easy to make the mistake of asking too many questions, collecting information that you don’t even need, and ending up in a situation where you’re not sure what the data is actually telling you.
- Crafting a focused survey that asks the right questions in the right way and collects the right information takes time. Just like design prototypes, UX research surveys should be iterated with project teams and stakeholders before being put out into the world.
- Before you even start creating a survey in an online tool, take a step back and make sure you’re clear about what you actually want to find out — not what you could ask users.
- It’s also worth remembering that what people say and do can be different. So don’t completely rely on self-reported data from users — direct observation is often better to gauge UX than what people tell you.
- It’s tempting to be lazy and not do any research beyond surveys. But while you can collect qualitative data from surveys, this often lacks context and detail and has limited value compared to what you get from qual-specific techniques, e.g. user interviews.
When to use it
Surveys can be used for all kinds of UX research — from early discovery to post-launch to get feedback about the live product. They can also be used to complement and recruit to other types of research, e.g. you can invite people completing your survey to leave their contact details so you can contact them about taking part in interviews or usability testing.
17. Task analysis
What is it?
A research approach used to understand how tasks are performed.
How does it work?
- Task analysis is all about understanding how people do things: where do they start? What steps do they take? What order do they do things in? And what information do they need to complete the whole process?
- Learning how users perform tasks can be carried out through a bunch of different activities, including interviews, contextual inquiry or diary studies. By using different techniques to understand the same task, a true picture may emerge.
- The whole idea of task analysis is for UX researchers to get a feel for the current way things get done. Taking a fresh, independent and objective look at a task process can highlight things like inefficiencies, bottlenecks and single points of failure.
- Using a systems thinking mindset, UX designers are then able to propose solutions for new workflows — achieving the goal of the task, but in a more effective and efficient way.
When to use it
Task analysis can be done at any time. It can be useful to find problems a new product or system could solve. But it could also be used on ‘business-as-usual’ processes, which often grow in complexity and inefficiency over time.
Note: I’ve included task analysis as its own entry as it’s possible to analyse a task in a much simpler way than carrying out a whole contextual enquiry study, or other UX research methods. For example, it could just be a simple conversation between colleagues that highlights an obviously more efficient process.
18. Tree testing
What is it?
A method used to validate the design of an information architecture (IA).
How does it work
- Tree testing involves asking users to find specific information within an IA using a software tool like Maze.
- The purpose of tree testing to establish whether the information categorisation, labelling and hierarchy is intuitive — essentially, is it easy for users to find what they’re looking for?
- Tree testing can be carried out at any point in a product’s life cycle, and requires only the tree itself and the set of tasks — no coding or user interface (UI) design is needed.
- You can also carry out tree testing to compare two different approaches, i.e. two versions of a page tree for a website.
- All tree testing experiments produce quantitative data. The results can be visualised within the online testing software itself, or exported to analyse manually.
When to use it
Tree testing is about validation, not exploration. You should use it once you’re already pretty confident in your IA — perhaps from doing prior card sorting research.
Note: Page tree and site map are basically the same thing in the context of IA and tree testing research.
19. Usability testing
What is it?
A method where users test how easy it is to complete tasks using a digital product.
How does it work?
- Usability tests can take place anywhere, including online or in-person. But it’s considered ideal to carry out tests face-to-face in a controlled environment like a a UX lab or some other quiet, calm space.
- A usability test typically consists of a facilitator and observer from the UX design side, and a participant (the user). Usability testing sessions typically last about 45–60 minutes.
- In the test, the facilitator will ask the user to try and complete different tasks via a UI, e.g. website or mobile app. These tasks could include anything from going through a step-by-step transaction to simply finding information.
- The facilitator’s job is to prompt the user to talk through their thought process, their expectations and frustrations as they attempt to complete tasks. They should avoid the temptation to help the user, and should ask a mix of open and closed questions.
- The observer’s job is to make a note of what the user does and says, paying particular attention to areas of confusion. This insight will highlight design changes that will improve the UX of the product.
When to use it
Usability tests can be carried out on existing products to understand areas of poor usability, or to test new products and systems as they’re being developed. How many users should you test with? It’s considered that 5 users is enough to discover most usability issues (after that you run into the law of diminishing returns) — however, there’s been some debate about this.
Always remember — and ensure the participant understands — that usability testing is a test of the design, not the user!
20. Wizard of Oz
What is it?
A method of simulating product responses to show how it could work.
How does it work?
- In a Wizard of Oz test, a UX researcher will manually simulate responses to user interactions to create the impression of it being a fully functioning product.
- Example: if researchers wanted to test user reactions to an AI chatbot, it’s not necessary to actually build a working chatbot before doing any testing. The user could interact with an instant messaging interface that looks like it could be a chatbot, and a researcher — perhaps in another room — types responses.
- Wizard of Oz is a smart way to quickly test concepts without having to code a complex system behind the design. It also allows UX researchers to completely change the ‘functionality’ during the same testing session (by the ‘Oz’ researcher simply changing the way they respond).
- This flexibility allows researchers to easily scrap ideas that don’t work — because it’s as simple as the Oz researcher not repeating that approach. Talk about rapid prototyping!
- The results of Wizard of Oz tests can suggest to researchers how an actual, functioning product or system might work best for users.
When to use it
Wizard of Oz testing works best when developing new, and often innovative, products that users may not be familiar with. While it can provide great qualitative data about the functionality of a prospective product, it’s also a useful technique to assess how users feel about the idea itself — and how likely they’d be to accept it and use it.
Conclusion
- User research is a core component of digital UX — it’s borderline essential to design products and services that solve user problems and meet their needs.
- While people can specialise in UX research itself, all self-respecting designers should have some knowledge, skills and experience of testing with users.
- The key with UX research is to get the right balance of different methods and amount of time, money and effort spent on research. An appropriate UX research plan will look different for every project.
- As a rule of thumb, match the scale of UX research you do to the scale and importance of the project, get a good balance of quantitative and qualitative data (to find out what’s happening, but also why), and quit when you hit diminishing returns (i.e. you’re not finding out anything new).
- The easiest methods to deploy on most projects are analytics, interviews, focus groups, surveys and usability testing. I consider these a sort of ‘core five’ research methods.
- Remember: you don’t need a huge budget or timeframe to carry out UX research — be flexible and adaptable and match your research plan to your resources. Even if you have zero budget and little time, there’s always some worthwhile research you could do — you might just have to get creative.
Return to Part 1
This is Part 2 of the playbook. You can read Part 1 here.
Sources
Badiu, R. (2023) UX research methods: glossary. Available at: https://www.nngroup.com/articles/research-methods-glossary/ (Accessed: 8 March 2024)
Becker, C. R. (2020) Learn human-computer interaction: solve human problems and focus on rapid prototyping and validating solutions through user testing. 1st edn. Birmingham: Packt Publishing, Limited.
Coleman, B. and Goodwin, D. (2017) Designing UX: prototyping. 1st edn. SitePoint.
Gordon, K. and Rohrer, C. (2022) A guide to using user-experience research methods. Available at: https://www.nngroup.com/articles/guide-ux-research-methods/ (Accessed: 8 March 2024).
Hay, L. (2017) Researching UX: Analytics. 1st edn. SitePoint.
King, R., Churchill, E.F. and Tan, C. (2017) Designing with Data: Improving the User Experience with A/B Testing. 1st edn. Sebastopol: O’Reilly Media, Incorporated.
Marsh, S. (2018) User Research: A Practical Guide to Designing Better Products and Services. 1st edn. London: Kogan Page, Limited.
Martinelli, S., Lopes, L. and Zaina, L. (2024) ‘UX research practices related to long-term UX: a systematic literature review’, Information and Software Technology. Available at: https://doi.org/10.1016/j.infsof.2024.107431
Nunnally, B. and Farkas, D. (2017) UX Research: Practical Techniques to Designing Better Products. 1st edn. Sebastopol, CA: O’Relly Media, Inc.
Rohrer, C. (2022) When to use which user-experience research methods. Available at: https://www.nngroup.com/articles/which-ux-research-methods/ (Accessed: 8 March 2024).
Sundt, A. and Eastman, T. (2019) ‘Informing Website Navigation Design with Team-Based Card Sorting’, Journal of Web Librarianship, 13(1), pp. 37–60. Available at: https://doi.org/10.1080/19322909.2018.1544873
About the author
Andrew Tipp is a lead content designer and digital UX professional. He works in local government for Suffolk County Council, where he manages a content design team. You can follow him on Medium and connect on LinkedIn.