The Solution

Communicate with Stakeholders

Involving stakeholders throughout the research process is crucial not only because they have a lot of important information that might help design the study but also because it would provide us a bigger picture of the product. Thus, before designing the study, I’d first interact with the stakeholders and ask them a lot of questions. For example, I’d ask:

• Data Scientists: Where do Youtube visitors come from? What are the characteristics of visitors who abandon the site watching only one video and those who watch additional videos respectively? What are the exit rates by different landing pages/education levels/income levels/other characteristics?
• Product Managers: Who are our target users? How important is the problem? What do you want to achieve through the research? How much time/budget do we have for the study?
• Designers: What are their assumptions of why users abandon the site after watching only one video?

What methodology to use?

There’re a lot of user research methods out there and choosing what method to use really depends on the goals of the study.

The reasons I choose remote moderated usability study are：
1) It allows users to perform the task in their natural environment, which reduces the effect of lab-based intrusions;
2) It allows me to see both quantitative data (e.g. time on task) and qualitative data (e.g. open-ended interview questions) and both behavioral data (e.g. click path) and attitudinal data (e.g. ease of task);
3) It’s quick and cheap compared to traditional lab in-person usability test

Who to recruit?

10–15 representative users from either out source panel or internal mail list:

Who to recruit really depends on who our target users are. In this YouTube case, we need to make sure participants who we recruit for the study must meet the standards of the target YouTube users. However, if from the website data before we design the study we found that people with a certain characteristic (e.g. age>45) are more likely to exit than others, then we need to recruit participants accordingly.

We must also be sure not to recruit people with certain characteristics: e.g. people who have never used internet or people who are experts in the field of user experience.

What tools to use?

UserTesting.com: There’re a lot of good user testing tools. UserTesting.com is one that I think work best for this study because it collects both quantitative and qualitative data and allows moderation; while UserZoom is another good testing tool but is more of quantitative oriented.

What to test?

Task: I’d set a well-defined task together with the stakeholders and ask participants to perform the task as they’d normally do.
E.g. “Let’s say you went to a Chinese restaurant yesterday and really love the Hot & Sour Soup. You want to learn more about how to make the soup by watching YouTube videos. Demonstrate how you access and use YouTube in this case. Click “Finish” when you believe you’ve shown us how you normally use YouTube.”

Interview: I’d ask users to explain their certain behaviors in the task. I’d also ask a set of pre-determined questions such as ease of task, overall satisfaction and net promoter scores.

How to analyze data?

Qualitative data such as participants’ comments or answers to open-ended questions can be analyzed by categorizing into different themes.

Quantitative data can be used to check if assumptions are true. For example, we can check if the assumption “users exit after watching only one video before they find recommended videos not relevant” is true by analyzing if users’ satisfaction scores are correlated with their perceptions of the relevantness of recommended videos. However, since it’s a 10–15 people user study, caution should be taken when using statistical terms.

How to report study results?

I’d first report descriptive statistics to stakeholders. For example, average time on task, average number of videos being watched, average satisfaction scores. Then, I’d focus on commonly occurring themes and provide actionable recommendations.

Whether the study is effective?

There’re a lot of ways to determine if the changes are effective. For example, if the goal is to increase the number of videos being watched by users, we can compare if the number of videos watched by per use increased after we make changes to the website according to study results. We can also compare the average time users spend on YouTube before versus after the changes.