Quantitative User Research Methods
What is Quantitative Research?
There are lots of User Experience Research methods. Some of them are great to produce data and insights while others support ongoing activities to get things done. It is important to combine the right things at the right time. A quantitative research method is used to collect data about user behavior, activities, experiences over a time when the user tries to accomplishes a task or goal. While conducting quantitative research, instead of gathering insights, we’re gathering numbers that describe some aspects of User Experience. These numbers are called as UX metrics.
Qualitative Research is another important method followed in User research. Read more about Qualitative Research.
Types of Quantitative Research:
There are many types of quantitative research. The widely used methods are:
- Surveys
- Analytics
- A/B Testing
- Quantitative Usability Testing
Surveys:
Surveys are one of the great user research methods because they’re often quick and relatively cheap. Surveys do not measure objective performance, they measure user perceptions. Surveys are based on User Perception and they are highly biased. Surveys can be considered as quantitative as well as qualitative, but both are subjected to response bias. There are three types of bias in survey results. They are:
1. Acquiescence Bias: The tendency to click “Agree” or say “Yes” to a list of survey questions is called acquiescence bias. For example: If you’re running a survey to check the experience of the application on a 1 to 5 scale where 1 strongly disagreed and 5 is strongly agreed, people tend to select agree or strongly agree most of the time. This will result in positively skewed results.
2. Social Desirability Bias: It’s no secret that we always like to project our best selves. This results in a response bias called social desirability bias. The tendency to over-report socially desirable behaviors and characteristics but under-report socially undesirable behaviors and characteristics. For example: Consider the survey in the below-attached image
To questions like these, respondents are more likely to underestimate the amount of usage of social media. Because excessive usage of social media is often considered socially undesirable.
3. Recency Bias: Recency bias is caused due to the short term memory of the human brain. The respondents will have a tendency to give more weight to recent experiences rather than their overall experience. For example: Consider the question from the below-attached image.
People tend to rate positively or negatively based on the most recent experience rather than the overall experience.
Response bias is the main reason that we need a large sample size of data. Large sample sizes help us to eliminate random thoughts and assumptions and ensures that our findings would be seen in our entire user population. Knowing that this response bias exists, it doesn’t mean we can eliminate them. However being aware that this bias exists, can help us better evaluate the survey results.
Know more about creating survey questions on the hotjar blog.
A/B Testing:
A/B testing is a design optimization technique that uses analytics data to determine which version has the most positive impact on the user experience. This is carried out by splitting the live users between different design variants in order to test their impact. If you’re changing the design, and you want to be sure that any effects you see are truly caused due to the change in design, consider performing A/B test. A/B testing compares the original design (i.e) Version A with an alternative version, Version B. The alternative version is modified with changes that you think will improve the user experience. A/B testing is carried out by randomly assigning real users to both versions and checking whether people act differently when they get a B version. Because all the user sessions are happening exactly at the same time and the exact same circumstances. From this data, we can be much more confident that the change in user behavior was actually caused by the changes you made in the design and not by any other external factors. This helps to choose which variants result in the best impact on conversations.
Analytics:
Analytics is seeing our users how they are using our site in real-time. Analytics data track exactly how users behave, showing you every click, tap, swipe, and scroll which is an incredible source of information. We can use tools like Google Analytics, that will track how many users arrived at the site, how did they arrived, through which medium, they arrived, what they are doing in the site, which page are they currently watching, how long they stay in the site, etc. Analytics is useful because they are very inexpensive.
Quantitative Usability Testing:
Quantitative Usability Testing tends to be very expensive when compared to Analytics. This can be conducted in person or remotely. Usability testing is nice although its expensive because you can control the conditions and its experimental. This is done by asking the users from the userbase to perform a certain task that they actually perform on the application or site. After conducting this testing with many users we can observe the results and find out what breaks down, where the user tends to drop off, how quickly they can complete a certain task, how do they feel while completing the tasks, etc. Even though this seems to be a bit expensive, we can get a bit of a richer picture of the usability of the site.
When to use Quantitative Research?
Quantitative Research methods are used to determine the priority or scale of the problem. To understand what proportion of the targeted users are impacted by the problem. You can use this analysis to compare the alternative solutions which have some gaps in addressing a particular problem. This can also be used to compare alternative design options. Benchmarking the UX is one of the important use cases for quantitative research, that means tracking the user experience overtime to make sure that you’re improving. Often this analysis is used to compare different versions of the same product/service or our product with competitor products. Quantitative Research will be helpful in calculating the ROI of a product or service. The major benefit of quantitative research is that it gives us the ability to determine whether the difference between the two numbers is statistically significant. To achieve this we need larger sample sizes of quantitative data. If we don’t collect the right amount of data, we may not have a reliable result.
Conclusion
User Research is an important phase of any project. If you’re going to conduct user research, make sure you’re doing it right. Basically, the user research returns two types of data — Qualitative and Quantitative and there are multiple methods in both types of research to get different types of data. Understand the best practices of those methods before using them in real-time. Before deciding on the user research methods first ask yourself these questions, What goals are you trying to achieve? Why you’re running this research? What you’re trying to improve in the exiting solution?
