Introduction
The purpose of this article is to explore various types of quantitative research in UX design methods and summarize their features, including advantages, disadvantages, and use cases because quantitive research is one of the most important factor in UX design. Before delving into the explanation of each quantitative research method, I would like to discuss the role of UX research and quantitative methods in UX design.
Why UX research matters for UX design:
- UX research means finding out what users need, want, and do. It helps designers make good choices and create products or services that work well and make users happy. UX research can make products or services better, faster, and more useful.
Two kinds of UX research:
- Primary research is when UX researchers get new data from users or people. It helps to learn what users think and do. Secondary research is when UX researchers use data that someone else collected, such as books, articles, or reports. It helps to get more information and see the big picture.
What primary and secondary research do:
- Primary research gets specific information for a certain question or goal. Secondary research adds more information to primary research. It helps to understand the problem or chance better.
Two ways to do primary research:
- Qualitative research is when UX researchers get data that is not numbers, such as words, videos, or sounds. It helps to understand ideas, feelings, or experiences. Quantitative research is when UX researchers get data that is numbers, such as time, errors, or scores. It helps to find patterns, averages, or links.
How qualitative and quantitative research help UX design:
- Qualitative research gives data that tells why and how users behave and feel. Quantitative research gives data that tells how much and how often users behave and feel. They help UX researchers check ideas or choices.
List of methods to be introduced in this article
Quantitative Usability Testing
What is this? : typical data and use
Quantitative usability testing is a method of evaluating the usability of a product or service by measuring how users interact with it. It involves collecting data such as task completion rates, error rates, time on task, satisfaction ratings, etc.
What can we find?
Quantitative usability testing can be used to compare different designs or prototypes, identify usability problems, and measure user performance and satisfaction3.
advantages:
- It provides objective and reliable data that can be used to make informed decisions.
- It can reveal hidden issues that may not be detected by qualitative methods or user feedback.
- It can help quantify the impact of usability improvements on user behavior and business outcomes.
disadvantages:
- It requires a large sample size to ensure statistical validity and generalizability.
- It may not capture the underlying reasons or motivations behind user behavior or preferences.
- It may not reflect the real-world context or scenarios that users encounter.
A case study of quantitative usability testing is the one conducted by Google to optimize the design of their search results page. They used an eye-tracking device to measure how users scanned the page and where they clicked. They found that users tended to focus on the upper left corner of the page and ignored the right sidebar. They also discovered that users preferred fewer results per page and more white space. Based on these findings, they redesigned their page to improve user experience and engagement.
Analytics
What is this? : typical data and use
Analytics is a method of collecting and analyzing data from various sources to gain insights and optimize performance. It involves using tools such as Google Analytics, Adobe Analytics, Mix panel, etc. to track user behavior, interactions, conversions, etc. on websites, apps, or other digital platforms.
What can we find?
Analytics can be used to measure user acquisition, retention, engagement, loyalty, satisfaction, etc3.
advantages:
- It provides a large amount of data that can be used to identify trends, patterns, segments, etc.
- It allows for real-time monitoring and evaluation of user behavior and outcomes.
- It enables data-driven decision making and experimentation.
disadvantages:
- It may not capture the quality or depth of user experience or feedback.
- It may not account for external factors or influences that affect user behavior or outcomes.
- It may require technical skills and resources to collect, analyze, and interpret data.
A case study of analytics is the one conducted by Netflix to improve their recommendation system. They used analytics to collect data on user preferences, ratings, viewing history, etc. They also used machine learning algorithms to analyze the data and generate personalized recommendations for each user. They found that their recommendation system increased user satisfaction, retention, loyalty, and revenue5.
A/B Testing
What is this? : typical data and use
A/B testing is a method of comparing two or more versions of a product or service to determine which one performs better. It involves randomly assigning users to different groups (e.g., A or B) and exposing them to different versions (e.g., A or B) of a product or service (e.g., website design, landing page, email subject line, etc.). The performance of each version is then measured by a predefined metric (e.g., click-through rate, conversion rate, bounce rate, etc.).
What can we find?
A/B testing can be used to test hypotheses, optimize designs, increase conversions, etc3.
advantages:
- It provides empirical evidence that can be used to validate or invalidate assumptions.
- It allows for controlled experimentation and causal inference.
- It can help optimize products or services based on user feedback.
disadvantages :
- It requires a large sample size and sufficient time to ensure statistical significance and validity.
- It may not account for confounding variables or interactions that affect user behavior or outcomes.
- It may not reflect the long-term effects or implications of each version.
A case study of A/B testing is the one conducted by Amazon to test the impact of free shipping on customer behavior. They used A/B testing to compare two versions of their website: one that offered free shipping for orders over $25 and one that did not. They measured the conversion rate and revenue of each version. They found that the free shipping version increased the conversion rate by 45% and the revenue by 25%. Based on these results, they decided to implement the free shipping policy across their website.
Surveys
What is this? : typical data and use
Surveys are a method of collecting data from a group of people by asking them questions. They can be conducted in-person, over-the-phone, or online. Surveys can use closed-ended questions (e.g., yes/no, multiple choice, rating scales, etc.) or open-ended questions (e.g., text, audio, video, etc.).
What can we find?
Surveys can be used to measure attitudes, opinions, preferences, satisfaction, etc1.
advantages:
- They provide a large amount of data that can be used to describe or compare populations or groups.
- They allow for standardized and consistent data collection and analysis.
- They can reach a wide and diverse audience at a low cost and time.
disadvantages:
- They may suffer from low response rates or non-response bias.
- They may not capture the complexity or nuance of human behavior or experience.
- They may be influenced by social desirability bias or other types of response bias.
Case study:
A case study of surveys is the one conducted by Pew Research Center to measure public opinion on various topics such as politics, religion, social issues, etc. They use surveys to collect data from representative samples of the population using various modes such as online panels, telephone interviews, mail surveys, etc. They analyze the data using statistical techniques such as weighting, regression, cluster analysis, etc. They report the results using charts, tables, reports, etc.
Eye Tracking
What is this? : typical data and use
Eye tracking is a method of measuring where and how users look at a product or service. It involves using a device such as a camera or a headset to record the eye movements and gaze patterns of users.
What can we find?
Eye tracking can be used to evaluate the usability, attractiveness, attention, emotion, etc. of a product or service (e.g., website design, advertisement, video game, etc.)3.
advantages:
- It provides direct and objective data on user behavior and experience.
- It can reveal subconscious or implicit aspects of user perception and cognition.
- It can complement other methods such as usability testing or analytics.
disadvantages:
- It requires specialized equipment and software that may be expensive or difficult to use.
- It may not capture the context or motivation behind user behavior or experience.
- It may be affected by external factors such as lighting, noise, distractions, etc.
A case study of eye tracking is the one conducted by Spotify to optimize their music streaming app. They used eye tracking to measure how users interacted with their app and what features they noticed or ignored. They found that users spent more time looking at the album art than the song titles or artists. They also discovered that users preferred horizontal scrolling over vertical scrolling. Based on these findings, they redesigned their app to improve user experience and engagement.
Heatmaps
What is this? : typical data and use
Heatmaps are visual representations of where users look or click on a product or service. Heatmaps can be generated by using tools such as eye-tracking devices or software that track mouse movements or clicks.
What can we find?
They can help identify which areas attract more attention, which areas are ignored, and how users navigate the interface3. Heatmaps in UX research visualize where users focus their attention and how they navigate through a product or service’s interface.
advantages :
- They provide direct and intuitive data on user behavior and experience.
- They can reveal hidden or unexpected patterns or insights that may not be detected by other methods.
- They can complement other methods such as analytics or usability testing.
disadvantages:
- They may not capture the context or motivation behind user behavior or experience.
- They may not reflect the real-world environment or scenarios that users encounter.
- They may require specialized equipment or software that may be expensive or difficult to use.
Case study:
A case study of heatmaps is the one conducted by Spotify to optimize their music streaming app. They used heatmaps to measure how users interacted with their app and what features they noticed or ignored. They found that users spent more time looking at the album art than the song titles or artists. They also discovered that users preferred horizontal scrolling over vertical scrolling. Based on these findings, they redesigned their app to improve user experience and engagement.
Funnel Analysis
What is this? : typical data and use
Funnel analysis is a method of tracking and analyzing how users move through a series of steps or stages in a product or service. Funnel analysis can be performed by using tools such as Google Analytics, Mixpanel, Amplitude, etc. that track user actions and events.
What can we find?
It can help measure conversion rates, drop-off rates, and identify bottlenecks or barriers3.Funnel analysis in UX research tracks the flow of users through a series of steps, measuring conversion rates and identifying potential barriers to user engagement.
advantages:
- They provide quantitative data on user behavior and outcomes.
- They allow for real-time monitoring and evaluation of user progress and performance.
- They enable data-driven decision making and optimization.
disadvantages:
- They may not capture the quality or depth of user experience or feedback.
- They may not account for external factors or influences that affect user behavior or outcomes.
- They may require technical skills and resources to collect, analyze, and interpret data.
Case study:
A case study of funnel analysis is the one conducted by Airbnb to increase their bookings. They used funnel analysis to track how users moved from searching for a listing to booking a stay. They found that many users dropped off at the last step of the booking process because they had to wait for host approval. To solve this problem, they introduced instant book feature that allowed users to book without waiting for host approval. They found that this feature increased their bookings by 2%.
Cohort Analysis
What is this? : typical data and use
Cohort analysis is a method of comparing the behavior and outcomes of different groups of users over time. Cohort analysis can be performed by using tools such as Google Analytics, Mixpanel, Amplitude, etc. that segment users based on common characteristics or attributes (e.g., sign-up date, location, device, etc.).
What can we find?
It can help measure user retention, loyalty, engagement, and lifetime value3.Cohort analysis in UX research compares the behavior and outcomes of different user groups over time, helping to identify trends, retention patterns, and engagement levels.
advantages:
- They provide longitudinal data on user behavior and outcomes.
- They allow for granular and customized analysis of user segments.
- They enable identification of trends, patterns, segments, etc.
disadvantages:
- They may suffer from low sample size or data quality issues.
- They may not capture the causality or correlation between user behavior and outcomes.
- They may require technical skills and resources to collect, analyze, and interpret data.
Case study:
A case study of cohort analysis is the one conducted by Facebook to measure their user retention. They used cohort analysis to track how many users who signed up in a given month returned to the platform in the following months. They found that their retention rate was declining over time, indicating that they were losing users faster than they were acquiring them. To address this issue, they introduced various features and initiatives to improve user engagement and retention, such as news feed, timeline, groups, etc.
Click Test
What is this? : typical data and use
Click test is a method of testing how users react to a single element or page of a product or service. Click test can be performed by using tools such as Maze, UsabilityHub, Optimal Workshop, etc. that present users with a static image or prototype and record their clicks.They involve asking users to click on a specific area or element and measuring their response time, accuracy, and feedback.
What can we find?
Click tests in UX research measure user reactions to specific elements, providing quick insights into user behavior and preferences for design optimization.
advantages :
- They provide quick and easy data on user behavior and perception.
- They can help test hypotheses, assumptions, or preferences.
- They can help optimize elements or pages based on user feedback.
disadvantages:
- They may not capture the context or motivation behind user behavior or perception.
- They may not reflect the dynamic or interactive nature of a product or service.
- They may be influenced by external factors such as screen size, resolution, device, etc.
Case study:
A case study of click test is the one conducted by Dropbox to test their landing page. They used click test to measure how users reacted to their value proposition and call to action. They found that users were confused by their value proposition and did not understand what Dropbox was or how it worked. They also found that users did not notice their call to action or did not know what to do next. Based on these findings, they redesigned their landing page to make it more clear, simple, and compelling.
Card Sort
What is this? : typical data and use
Card sort is a method of testing how users understand and organize information. They involve asking users to sort cards with labels or images into categories that make sense to them.
What can we find?
Card sort can be performed by using tools such as Optimal Workshop, UserZoom, UsabilityTools, etc. that present users with a set of cards and categories and record their sorting choices. Card sorts in UX research help understand how users mentally organize information, aiding in the creation of intuitive navigation and labeling for products or websites.
advantages:
- They provide qualitative and quantitative data on user behavior and cognition.
- They can help create intuitive information architecture, navigation, and labeling.
- They can help identify user mental models, expectations, and needs.
disadvantages :
- They may not capture the complexity or nuance of human behavior or cognition.
- They may not reflect the real-world context or scenarios that users encounter.
- They may require a large sample size and sufficient time to ensure validity and reliability.
Case study:
A case study of card sort is the one conducted by Spotify to improve their music streaming app. They used card sort to measure how users understood and categorized different types of music. They found that users had different preferences and expectations for music genres, moods, activities, etc. Based on these findings, they redesigned their app to offer more personalized and diverse music recommendations.
Positioning and roles of each method
Measurement: surveys
- These methods are used to measure the attitudes, opinions, preferences, and satisfaction of users in a product or service.
Analysis: analytics, funnel analysis, and cohort analysis
- These methods are used to analyze the behavior, patterns, and outcomes of users in a product or service.
Evaluation: quantitative usability testing, A/B testing, and eye tracking
- These methods are used to evaluate the usability, performance, and effectiveness of a product or service.
Optimization: heatmaps, click tests, and card sorts
- These methods are used to optimize the design, layout, and content of a product or service.
Conclusion:
In this article, we explored various types of quantitative research methods in UX design and discussed their features, advantages, disadvantages, and use cases. Quantitative research plays a crucial role in understanding user behavior, preferences, and experiences, and it provides valuable data for data-driven decision-making and design optimization. Each method serves a specific purpose in UX research, such as measuring user attitudes, analyzing user behavior, evaluating usability, and optimizing design elements. By utilizing these quantitative research methods effectively, UX designers can create products and services that better meet user needs, enhance user experiences, and achieve business goals.
References
:What is UX Research? | IxDF — The Interaction Design Foundation
: User Research in UX Design: The Complete Beginner’s Guide — CareerFoundry
: What is the difference between primary and secondary research in UX …
: Qualitative vs. Quantitative Research | Differences & Methods — Scribbr
: What Is Quantitative Research? | Definition, Uses & Methods — Scribbr
:Quantitative Research: Methods and Examples | SurveyMonkey
: Eye-tracking study shows how users scan Google search results — UX Planet
: How Netflix Uses Analytics To Select Movies — Dataconomy
: The Ultimate Guide To A/B Testing — Smashing Magazine : [About Pew Research Center — Pew Research Center](https://www.pew
What is UX Research? | IxDF — The Interaction Design Foundation. https://www.interaction-design.org/literature/topics/ux-research.
User Research in UX Design: The Complete Beginner’s Guide — CareerFoundry. https://careerfoundry.com/en/blog/ux-design/the-importance-of-user-research-and-how-to-do-it/.
User experience research vs user experience design — UsabilityHub. https://usabilityhub.com/blog/user-experience-research-vs-user-experience-design.
What is UX Research and Why is it Important? | Net Solutions. https://www.netsolutions.com/insights/what-is-ux-research/.