The Secret Weapon of User Research: Experience Sampling

Customer Discovery is a well-known process, but often one great method is missing: The Experience Sampling Method (ESM).

Customer Discovery is the first step of the Customer Development process developed by Steve Blank. The goal is to define clear hypotheses which are then validated through observation, interviewing, and prototype testing. There are a lot of articles and books about how to observe and interview people, and how to do good prototype testing. But one very useful method is often missing: Experience Sampling.

What is Experience Sampling?

The Experience Sampling Method (also called ecological momentary assessment (EMA)) was developed by R. Larson and M. Csikszentmihalyi. It has become popular since the 1980s in large part due to the positive psychology movement, where it was used to find correlations between happiness and activities.

Experience Sampling helps you to understand which factors are most important regarding a certain behavior.

The method is simple: the same questions is asked various times. This can vary from several times per day to once every couple of weeks. Though asking the same question again and again you will see how the experience varies. This will help you to understand the factors which are related to these variations. When you know which are the most relevant factors you know where to focus in your observation and interviews.

This doesn’t replace observing and interviewing people but it helps you to set a focus on the important things.

Good Questions

The question(s) should be about repeated behavior. Don’t ask yes/no-questions. Don’t ask quantitative questions. Don’t ask about average behavior or behavior in general. Be specific and add a time-constraint („last time“). Don’t ask about opinions. Clearly define when the questions should be asked — this can be time-based or context-based.

Example: What did frustrate you when you wrote your last Medium article?

Number of Participants and Number of Answers

You want between 500–1000 answers because less is not enough data and more is often too much data to handle. Assume that one-third will be lost due to useless answers, duplicates, etc. That means that the number of participants depends on how often you ask the question. To figure out how often you should ask the question, think about how often the behavior might happen and then ask less times per day/week.

Recruiting

Participant recruiting is the bottleneck of every user research, so start early. To make sure your participants fit into your target audience, create a screener survey (e.g. with Google Forms). If you want people who are active on social media, don’t ask something like „Do you often post on social media?“. Ask instead „How often did you post something on Facebook, Google Plus, or Twitter in the last 7 days?“. This is specific, benchmarkable and with a time-constraint.

Execution

Choose the right medium to deliver the question to the participants. Today one of the easiest ways is to use WhatsApp or Facebook Messanger. If you have more than 10 participants you might want to automatize the process (e.g. via IFTTT).

Always start with a pilot-test first to see if there are any problems or struggles in the process. Watch incoming results and adjust if necessary.

Analysis / sense-making

Your main work is to classify the data. Therefore, think about groups before you launch the study. Start to improve your classification as soon as the first responses come in. Your goal is to find pattern and correlations. Visualize the data and discuss in with the whole team. Based on those patterns create behavioral insight statements (hypotheses) („People seem to … because …“) which should be validated later (e.g. via observing and interviewing people).

Ressources

„Validating Product Ideas“ by Tomer Sharon — http://www.leanresearch.co/book/

Google I/O 2014 — Don’t Listen to Users, Sample Their Experience! — https://www.youtube.com/watch?v=v1KKsLukIBE

5 Tips for Smartphone Experience Sampling & Ecological Momentary Assessment — http://de.slideshare.net/MetricWire/5-tips-for-smartphone-experience-sampling-ecological-momentary-assessment?next_slideshow=1

THE SCIENCE OF EMA & EMI — https://www.lifedatacorp.com/the-science/

The Experience Sampling Method — http://www.springer.com/cda/content/document/cda_downloaddocument/9789401790871-c2.pdf

Final thoughts

Discovering what people really need, something that other companies haven’t noticed yet is a key success factor for startups. The Experience Sampling Method can help to unveil such secrets. It can help you to set the right focus before you go on a field research trip. It also can help you to validate if the behavior you noticed while observing and interviewing people is really an important behavior.

Enjoy the post? Please click the ♡ below.