How to run Experiments for Startup Idea Validation
How to set up Experiments in the Early Stages of your Product or Startup to Learn as much as you can about the Problem to Solve
When you come up with a new idea for a product, startup, or side project, it can be super tempting to rush on and try to develop the idea, build an MVP, and launch it. But, as you probably know, that is also super risky. You’ll spend hundreds of hours building something that people might not like or need.
The smart thing to do is to go the Lean Startup way and run experiments to validate your idea and what you’re building along the way, getting input from your potential customers and users, and working towards a product they’ll love.
The type of experiment to run depends highly on how far your idea is developed and what you’re trying to test. There are (at least) three different types of experiment you can run, depending on your startup level.
- Level 1–2: Experiments for Idea Validation
- Level 3: Experiments for Problem-Solution Fit
- Level 4–5: Experiments for Product Market Fit
Experiments for Idea Validation
In the first stages of developing a new startup or product, you’ll need to make sure that there is an actual problem that your idea solves.
Popular terminology calls this ‘idea validation’, but you’ve got to be a little careful with that term. ‘Idea validation’ opens the door to confirmation bias, as it implies you’re looking to see if your idea is ‘correct’. In my experience, early ideas almost never are. Ideas start out rough, lopsided, undeveloped, and based on a very limited experience (your own).
Would you rather be correct? Or successful?
It isn’t really interesting to see if your original idea is ‘validated’, either. It is much more interesting to see if your idea connects with a problem real people experience in the real world everyday. If you can find such a problem, and can adapt your idea to connect to it, your idea might become a real thing. If not, it is just a waste of your time to work on it.
So, in this stage, rather than looking at your undeveloped idea with a true/false mindset, you should go in discovery mode. It’s about finding new information connected to the problem people experience. You need to fall in love with the problem and learn everything you can about it.
Forget about your solution
Perhaps your original idea had a specific solution in mind to solve that problem. In this stage, forget about that solution for now. It doesn’t matter at this point how you are going to solve the problem. It matters if the problem exists.
You also want to know what kind of problem you’re looking at. If there isn’t a true need for people to use your solution, if there isn’t a problem your idea solves for them, then you won’t overcome their natural inertia. People won’t adopt your solution. They’ll just continue living their lives as they did before.
- Pains: Problem that people experience every day, and that they can’t avoid or get around except at high cost or effort.
- Gains: Needs that people may experience, but they can survive without it. Something that incrementally improves their quality of life.
- Latent needs: New possibilities that people don’t know they need yet. An example: mobile phones were a latent need for most people in the 90s. Back then, most people simply could not imagine all the new use cases they use their phone non-stop for today. People simply don’t know they need it yet.
Of these categories, pains are easier to turn into a product, and latent needs are the hardest. For a latent need, you’ll probably have difficulty to find enough people that ‘get it’ early on to validate your results.
How to run an Idea Validation Experiment
Because you’re trying to uncover all kinds of qualitative information at this point, it is harder to run a clear-cut experiment where you test a hypothesis and based on the resulting data decide if the hypothesis was validated or not.
The information you get back will likely be vague, qualitative, and very much open to interpretation.
Still, defining risky assumptions and setting up an experiment with a clear hypothesis makes good sense. It helps you to focus what you’re looking for. By defining how many people you want to talk to, and how many observations to make, and what you’re looking for exactly in these observations, it is way easier to ward off confirmation bias. Just be prepared for a load of interpretable results. In this stage, a diversity of answers is good. (It’s also not that big a problem if you’re not seeing huge numbers of respondents, you’ll need to double check what comes out of this stage later with significant numbers. Better to do that when you know what to ask them.)
Examples of risky assumptions for idea validation:
- “People really have this problem”
- “People experience the problem on a daily basis”
- “People really care about this problem”
- “People would like to have a (better) solution”
- “It is a pain, not a latent need”
Example hypotheses for idea validation:
- “We can find 10+ people that experience this problem and interview them inside 3 days.”
- “More than 70% of 20+ interviewed people experience this problem every day.”
- “More than 70% of 20+ interviewed people say the problem costs them over $100 per year.”
The tools to use in this stage are primarily a good interview sheet with questions, and secondarily tools such as the Persona and Customer Journey to make sense of the results.
Example interview questions for idea validation:
- How many times did you experience the problem the last year?
- Can you tell me about the last time you experienced the problem?
- How did it make you feel?
- What happened or didn’t happen because of it?
- Do you have a workaround? Can you avoid the problem?
- What do you already do to prevent or avoid the problem from happening? Can you describe how you did that the last time it happened?
- How do you feel about the workaround?
Where to find your respondents
One of the most important aspects of running the experiment is where to find your respondents. This very much depends on the kind of idea you have, and the problem you want to solve.
The key is to find people that are from the actual target audience — or what you think your target audience is at the moment — as possible. Friends and family are great to test your interview questions with, but they won’t give you the information you need.
Think about the places (online and offline) where people experiencing the problem gather. What are the channels they use? What are the locations they visit? That’s where you need to be to interview them.
If you already have access to a channel (e.g. an existing product, a well-visited website, a partner channel) with people from your target audience, all the better. In fact, for many startup ideas, not having access to people to interview and run experiments with is a real drawback.
Interpreting the Results
When looking at the diverse and qualitative results, it can help a lot to differentiate them into categories:
- Quotes. What respondents actually said, and the types of words and descriptions they used.
- Perceived problem. How respondents described the problem and circumstances with their opinions, perceptions, and assumptions.
- Perceived needs. What respondents think they really need (what were they trying to achieve? What was their Job to be Done?). What other needs did they express? What solutions did they volunteer?
- Behaviour. The actual behaviours and activities respondents performed in the past that are related to the problem.
- Observations. What you as interviewer observed while interviewing.
- Conclusions. What in this collection of results impacts the idea (positively or negatively)
- Next steps. What other experiments can you run to get more information? What new ideas for (partial) solutions did you get? What further questions do you have about the problem?
Build your Mental Model of the Problem
Once you have your interpreted results, try to reconstruct customer journeys and persona’s. Even better: construct customer journeys together with some of the respondents. These journey maps and personas will serve as the basis of your mental model of the problem.
Iterate!
Keep reiterating your steps for idea validation until you don’t get so many new big surprises in the answers anymore — or until you get a big enough signal that tells you you’re on to something.
Keep experimenting!
PS. Been there, done that for Idea Validation? Have a look at validating Problem Solution Fit or validating Product Market fit.
Experiment Cookbook
If you want to learn more about designing and running experiments for Product Market Fit, Idea Validation, or Problem Solution Fit, check out the online course ‘Experiment Cookbook’! 9 Modules complete with 25 detailed ready-to-run experiment recipes.
This story is published in The Startup, Medium’s largest entrepreneurship publication followed by +436,678 people.
Subscribe to receive our top stories here.
