The Quick and Dirty Guide to Validating Your Startup Idea

How to design and conduct a meaningful lean startup experiment

Hi everyone, Alex here. At Launch Academy, I am lucky enough to surround myself with so many talented early-stage entrepreneurs. After four years of mentoring 1000+ entrepreneurs and building six startups, I’ve consolidated my learning and developed a framework that you can apply to your next startup idea.

TLDR:

  1. Fully understand the problem you’re solving
  2. Know who your early adopters are
  3. Make sure you are solving the problem for the right people
  4. State your hypothesis and create a test plan
  5. Design your test to validate your solution
  6. Set your expected metrics
  7. Compare your expected metrics with your observed metrics and make a decision
  8. Sign up to our free email course and receive a curation of startup resources that I’ve collected in the past 5 years

Step 1: Have an in-depth understanding of the problem

Most startup ideas come from an observation of a pain point that is personal to the entrepreneur. Therefore, it is easy for entrepreneurs to fall in love with their ideas and dive into “solution-land” immediately without making the effort to understand the full scope of the problem. However, it is critical because by doing so, you will avoid the number one mistake that startups make, which is building something that nobody wants.

“Pitch the problem, not the solution”- Dave McClure

To grasp the full scope of the problem, you should do the following:

  1. Conduct 20–25 problem interviews to get qualitative results so you can learn about what the problem is, how big of a problem it is, and who are having these problems.
  2. Ask the 5 Why’s to get to the root cause of the problem.
  3. Do a competitive analysis and understand the alternative solutions to your problem and how people feel about using these alternatives.
  4. Synthesize your findings from your interviews and design a short survey to validate the problem in a quantifiable way. Typeform is a great tool for this. You can reference Mike Fishbein’s Ultimate List of Customer Development Questions.

The above comic depicts a typical scenario that happens in most companies. This misalignment happens because people don’t have the problem anchored down so they ended up building something entirely different from what the customers wanted.

Step 2: Create a predictive persona

You probably heard it a thousand times that it’s important to map out everything you know about your customer: their age, sex, geographic location, income level, occupation, discretionary budget, type of car they drive, type of media they consume, brands they love and etc.

And chances are, you end up with something like this:

In theory, personas should allow us to better understand our real users so we know exactly how we should design our product and market it to them. Unfortunately, most people end up creating useless personas because they don’t result in actionable takeaways. Personas often times are projections of what we hope our users would be like, but they end up being nothing but fictional characters.

On the other hand, a predictive persona identifies the things that make people want to be a customer and uncovers any underlying anxieties or motivations surrounding their character.

As Laura from Invision points out:

a predictive persona is a tool that allows you to validate whether you can accurately identify somebody who will become a customer. The question you should be asking yourself isn’t “If I interviewed a user, would this describe her?” The question should be, “If I found a person like this, would she become a user?”

This becomes incredibly powerful because as you gather more data and insight, your predictive persona will eventually reflect real people you can find in the real world. This helps you narrow down your marketing efforts and could potentially save you a lot of guesswork and money.

  1. Recruit people who fit your predictive persona
  2. See if they convert into customers
  3. Repeat step 1 and 2 until you have an accurate persona-customer fit
  4. Repeat process for each customer segment

Step 3: Find your problem-persona fit

Repeat step 1 and 2 until you have a solid understanding of what the problem is and who are you solving it for. This is called a problem-persona fit. Here are a few examples of positive signals:

  • 80% of your interviewees rank the problem you are trying to solve as one of the top three pain points that they currently experience with
  • 100 people who fit your predictive persona ended up signing up to your landing page
  • 80% of your interviewees have rated the problem as a 9 or above magnitude in terms of pain
  • 10 potential customers that are ready to pay you to solve this problem for them

Step 4: Develop a product hypothesis

To put it simply, a hypothesis is an assumption that can be clearly proven wrong. For example:

“I believe restaurant owners will use our lightweight video resume app at least twice a month to hire servers quickly and they will convert to paid subscriptions after a 30-day unpaid trial because our product helps them hire 50% faster.”

This is called a product hypothesis because you are testing your assumption whether or not your intended audience will use your product. So let’s dissect our product hypothesis into a simpler formula.

Product Hypothesis = I believe [target market] will [do this repeatable action/use this solution], which will [result in expected measurable outcome] for [this reason]

A good product hypothesis:

  • is falsifiable, which means it can clearly be proven wrong
  • is written down
  • contains metrics that can be tested and measured

Step 5: Design your minimally viable product (MVP) or test

I define a startup as “a series of experiments designed to search for a repeatable and scalable business model”. And we do experiments because they keep us honest about the problem we’re trying to solve. In every experiment, there is a test, or in this case a minimum viable product (MVP).

“The minimum viable product or MVP is that version of a new product which allows a team to collect the maximum amount of validated learning about customers with the least effort”- Eric Ries, The Lean Startup

The key thing to highlight here is validated learning. Validated learning is what propels you forward with your decision making. After each experiment, you should have a good idea of what your next step is.

Remember, MVP is a strategy, and not a one-time thing. You can have multiple tests or MVPs to get to the validated learning you need to proceed to the next experiment.

Christopher Blank wrote a great guide on 15 ways to test your MVP here. Check out which strategy is more suitable for your stage.

Step 6: Define your expected metrics

To test the validity of your product hypothesis you must first establish the expected metrics that you are going to measure. In the video resume app example, we’re going to measure:

  1. The frequency of app usage per user
  2. The ratio between the number of servers hired in a month with using the app vs a benchmark average number of servers hired in a month without using the app
  3. The percentage of user who converted into paying customers after a 30-day trial period

Metrics keep us honest so we’re stay objective about our hypothesis. It’s easy to lie to ourselves about how great our ideas are, but if the data shows something different, you must make a decision on whether to pivot or keep going.

Step 7: Compare expected metrics with observed metrics and make a decision

When you compare your expected metrics with observable metrics, you will encounter the following scenarios:

Scenario 1: My observed metrics is nowhere near my expected metrics

In this scenario, your initial hypothesis is clearly invalidated. However, this may be a good opportunity to learn why that is. Perhaps your intended audience is not the right audience for the product. Perhaps you will uncover a new set of pain points from your customers. Whatever the case may be, you will have gained the insight you need to either pivot or quit.

Scenario 2: My observed metrics fell 40% or more short of my expected metrics

In this scenario, you must assess why your metrics fell short. For example, if you were expecting 1000 users to behave this way and only 600 did, you must investigate why. Did you set unrealistic expected metrics? Was the idea invalidated? Were you targeting the wrong audience? Do you need a bigger sample size? These are all the questions that you should be asking yourself before making a call. Chances are you probably gained new insight to the type of persona you should be targeting and would require more discovery with this specific customer segment.

Scenario 3: My observed metrics meet or exceed my expected metrics

Here, you have a positive signal that your users are behaving exactly or better than the way you anticipated. This could be the results of establishing realistic and obtainable metrics. However, it could also reflect low expectations so the key here is to find a balance and define what’s reasonable. When you receive a positive signal, you should replicate the experiment and roll out to a bigger sample size so by the end of the experiment, you are confident that a % of your users will convert into customers and behave the way you want them to.

Depending on the nature of your startup, here are some examples for what positive signals may look like:

  • Marketplace: When you have at least 50 transactions between a buyer and a seller over a period of one week
  • SaaS: When you have at least 20 beta customers (paying or not paying)
  • E-commerce: When you have at least 50 customers buying your product in a month
  • User-generated platform: When you see at least 20 user-generated content being posted every day
  • Mobile App/Game: When you see 1000 users who came back and use the app at least once a week

Last words of wisdom

A great experiment is one that is additive. By executing this framework, you will produce more accurate data that will inform better business decisions. The learning you accumulate from each experiment will add up and eventually guide you to a problem-solution fit. Then you will have a solid foundation to work towards a product-market fit, but that’s a discussion for another day.

Thanks for reading this far! If you’ve learned something from this article, please hit the recommend button so this knowledge can be shared with other early-stage entrepreneurs. If you are looking to learn more about this framework, we’ve produced a free 10 chapter email course that covers these topics in far more detail. Click here to check it out.


//About the Author:

Alex Chuang is the Co-founder and Chief Strategy Officer at Launch Academy, Vancouver’s leading tech incubator. Alex is a serial entrepreneur, UX designer and growth hacker.