From the Experiment Cookbook
Experiment Recipe: Exit Poll
Get fast customer feedback for Problem Validation
About this experiment
The ‘Exit Poll’ experiment aims to get a reaction from test subjects just at the time they have experienced the problem (or at least, that’s what you assume). If you want to know how people experience their commute, ask them when they come out of the train station, or park their car. If you want to know what people think about when they are purchasing a vacuum cleaner, ask them when they just bought one.
Most often, an exit poll experiment takes the shape of an interview. You’ll find a suitable location frequented by people you think experience the problem, and try to approach people there, hopefully convincing them to do a quick interview. In the interview, you’re most interested in the experience they just had, and what they think about that.
Datasheet
- Experiment: Exit Poll
- Type of experiment: Mixed (exploration or validation)
- Benefits:Easy to set up, fast
- Dangers: You need a good location to reach test subjects, it can be hard to get test subjects’ attention
- Use this experiment to: Quickly get feedback from a larger group of people about how they experience a problem, get first feedback if people are interested in your idea, get some qualitative data
- Time and resources required to set up: You’ll need a bit of time and manpower to pull this off. You’ll probably need a day to set it up, and a day to run it.
- Evidence level: Weak. You’ll get information, but you won’t easily get a commitment from people you talk to on the street. People self-select: they determine they want to spend time talking to you, which can skew results.
- Method: A good way to do it is to approach people in a busy location and ask them questions (or have them take action, see the hack below)
- Prototype: Interview questions, something to draw their attention
Step by Step
Step 1. What do you want to explore vs your riskiest assumption
If you’re approaching this as an exploration experiment, find out what you want to know about. What are your assumptions about how people experience the problem? What do you expect they will tell you?
If you’re approaching it to validate, you need to pinpoint your riskiest assumption. Check the default riskiest assumptions for the stage you’re in first, and use the Riskiest Assumption canvas to make sure you have the right one.
Next, build your hypothesis. Find out how many people you think you can approach reasonably, and how many you predict will answer positively.
Tip: To get an idea of the amount of people you can reach, use your calculator.
Take the amount of time it takes to get through your questions (typically this should be 2–3 minutes max for this experiment), and multiply that by four. You’ll need time to convince your next subject to be interviewed. Take the time you want to spend in the location, and divide that by the time per interview. Multiply by the number of teams, and that’s your upper bound.
Example: We did an ‘exit poll’ experiment at WebSummit 2018, where we talked to startup founders. We calculated as follows:
- Teams: 2
- Estimated time per interview: 5 minutes
- Time to find a test subject: 10 minutes
- Time on venue: 8 hours per day
- Days on venue: 4
- Total interviews per day max: 8 * 2 * (60/15) = 64 max.
- Total interviews for four days: 256 max.
In the end, we managed to talk to 92 startups in total. We underestimated how much doing so many interviews would wear us down, and successful interviews sometimes were more than 15 minutes long. Also, because we were only two people, each of us had to interview and write down the answers at the same time, which was impossible to do. Between interviews, writing down the information took time as well.
(Read more about this experiment in this post)
Step 2. Prepare
If at all possible, scout out the venue at a time slot similar to when you want to do your interviews. How many people are there? What are they doing? Are they in a hurry? And, also quite invaluable, do you have any competition?
It’s tough to get people to stop and talk to you when you’re in a spot where passers-by are used to being harassed by people passing out flyers, selling in the street, or doing other surveys. If you see other people do this, think about getting another location, or think about a way to signal that you are not selling anything when you do your experiment.
Best thing to do in preparation is to try to approach someone and note how they respond.
Create a Google Sheet for you and your team to fill in, placing all questions vertical and answers horizontally, so it will be easy to compare answers later.
Tip: If you’re going to count certain behaviours or answers, it may be easier to print out your sheet and keep a tally, or to use a tally counter on your phone, such as Tally 2. This can make it much easier to keep track of how many times different answers or actions occurred.
Step 3. Create a script with questions
The next step is to come up with suitable questions. Focus on questions about their recent experience. Add the questions in the sheet. Also, for each question, think about what answers you’ll be looking for to validate your assumption. What counts as a ‘positive’ answer for you?
Example: Questions asked to public transport commuters at the train station
- How was your journey today?
- When did you start your journey today?
- How often do you travel by train?
- How does your experience compare to other times?
- What did you love / hate about it?
- Would you recommend going by train to others? Why (not)?
It can be difficult to approach people and talk to them. It is socially awkward. It can help to fabricate an ‘excuse’ to make approaching people easier. Anything that draws (positive) attention can work. Think of a good smile, brightly coloured t-shirts, or something else that makes people curious. Giving away something for free can work wonders as well. If you can come up with something that draws a crowd of people, you’re going to have an easier task to interview them.
Tip: When you come up with something, it can help to make it look not too shiny and well organized: people will be suspicious you’re trying to sell them something. Make it look fun and friendly.
Just don’t stand around with a clipboard and fear in your eyes, that’s a sure way of making people hurry past you.
Step 4. Run the experiment
When it is time to run the experiment, brief the team well. Make sure you arrive at the location early, and that you bring enough refreshments and drinks to keep everyone going. When possible, use a car parked nearby or a table at a coffee place as a ‘home base’, where interviewers can take some rest and discuss recent findings. Keep track of how fast interviews are coming in, and take action if it is not going according to plan.
Tip: It can be difficult to get started. If you’re not doing this every day, especially the first few approaches will be difficult. Tell everyone this is expected, and promise an award for the person that first breaks the ice. Framing it as a game reduces a lot of the tension.
Step 5. Interpret the data
Use the Experiment Outcome canvas and your sheet to go over the results. Was it easy to approach people? How did team members experience it? What were the most surprising results?
First, check if you have enough data. Did you meet the quota you set in the hypothesis? (Note: in the Experiment Cookbook creating your hypothesis and designing your experiment is explained in detail)
Next, score the answers and see if your score validated the assumption. Because this typically still is a low-n experiment, take the outcome with a grain of salt if your signal was so-so.
If you find the experiment did not get a clear outcome or if you did not have enough data, think about extending the experiment or running it in a different location or time slot. What could you do differently to make your signal clearer? What was the single question that gave you the most information?
Hacks
A great hack that turns this experiment into a quantitative experiment, and makes it much easier to get people to contribute, is to set up two ‘gates’ passersby can select to walk through. Put them a little out of the normal route, so people have to go through them deliberately. Mark each with a clear one-line statement, and count people going through the gates.
We did this to find out what people thought of their commute with the Dutch Railway, at the exit of the train station. We set up three traffic cones, and market one gate with “I love my commute” and the other with “I hate my commute”. About 50 people deliberately walked through the gates in half an hour, most of whom said they hated their commute that day. We asked a few of the test subjects follow up questions, if they were open to that.
Warning: this experiment ended for us because we got kicked out of the train station. Make sure you find a spot where you don’t get expelled, or ask permission :)
Another hack a team in an innovation workshop came up with is to use an object to make people curious. The subject the team wanted to learn about was how people transport groceries to their urban home. They came up with a (scrappy) prototype of a robot that can carry your groceries for you. We ran the workshop in-house in a large corporate, and they decided to try and stop people going to the in-house cafe for lunch to ask them questions. People definitely were curious about this cardboard-and-plastic robot prototype, but when they were on their way to have lunch they didn’t stop to talk. On their way back, however, it was much easier. They obviously had been thinking about this out-of-the-ordinary sight over lunch and wanted to know more.
This post is based on content from the Experiment Cookbook, with over 20 detailed recipes for experiments for idea validation, problem solution fit, and product market fit. What is your favourite experiment? Let me know!
Keep experimenting!
