Scientific Approach to Building a Startup: Experiment Like in a Lab.

As the CEO of HotelQuickly, I want to be able to decide with confidence what kinds of projects and MPVs to fund, and which to cut. Focus is everything.

As part of my resolution to master one topic a month, I decided to study how to determine product-market fit (PMF) and scale the product once it’s ready to go.

There is a lot of clutter around growth hacking, automation, and A/B testing.

So I wanted to go deep on the subject and be able to apply what I’ve learned to propel the company further.

Set Up The Lab

To start, I took the framework of the 10 steps to accelerated learning I highlighted in my previous post.

• I set constraints: I had one month to master the subject.
• I made my motivation clear: I wanted to understand the fundamental principles of applying the scientific method to a startup’s two phases — PMF and building the growth engine.
• I set a specific, well-defined goal of being able to apply my learnings to HQ within one month.
• I would emulate the experts, read a minimum of one book and five blog posts, listen to relevant podcasts, and take Udemy courses.
• I asked myself open-ended questions about what I wanted to learn.

Some of the questions were:

Learn-Build-Measure

Startups have ultimately two stages: 1) “Customer Validation” (Search: pre-PMF) and 2) “Scale” (Execution: product channel fit and growth).

Contrary to popular belief, the much-celebrated “just do it” approach to building businesses is not the most efficient way to progress through those stages.

The alternative? Since the early 20th century, the scientific method has been applied to management and has accelerated the growth of today’s industrial giants. It’s the application of the same core principles of the feedback loop “build-measure-learn” that started the “lean-startup” school of thought.

“Lean start-up” approach

This method brings clarity to the process while ditching the nerdy jargon. You start by forming the thesis of your business, e.g. what problem you’re trying to solve, and for whom. Then you define the right metrics that will indicate whether the thesis is correct.

Next, you put the right team in place to validate the thesis with an array of experiments.

“The success of a company is a function of how many experiments it does per year, per month, per week, per day.” Jeff Bezos

Once your fundamental business hypothesis is right, you reach PMF and use the same approach to scale the company.

Instead of giving you a textbook rundown of the lean start-up approach, here are some learnings that I found interesting:

First Things First: Find Your Product-Market Fit

Make sure you have a solid product-market strategy before you start thinking about scale and growth.

Keep operations manual until you learn enough about your customers.

Especially in the early days of a startup, you don’t have enough data to reach statistical significance of your tests, so you want to focus a lot more on qualitative elements of the test and learn approach.

1. Predictive Approach

Begin with the end in mind and reshuffle the order of “build-measure-learn” to “learn-measure-build.” First figure out what you need to learn to build the product, then what needs to be measured, then build the MVP.

2. Hypothesis

Validate the startup idea with a measurable customer hypothesis. Do not start by building a product. You want to ensure that the customer hypothesis you create is falsifiable (“nobody wants XYZ”). List all your assumptions.

For example, for localisation of your product to different markets consider:

• “What if the customer doesn’t care about the sleek, minimalist, design as much as you do? What if the design doesn’t fit customer behaviour?”

3. Metrics

Everyone in your organisation should rally around a few key metrics. This can prove very tricky in the PMF stage.

There is a lot of noise about “traction” with vanity metrics that drive “growth” but ultimately have nothing to do with whether you are reaching PMF.

The false reinforcement you can get for driving “traction” makes it very easy to slip into the money-burning machine with crappy unit economics and no product-market fit.

Source: Coelevate

Ultimately, true product-market fit boils down to four critical metrics:

• Conversion moment: Your customer pays for the product/service you offer. Ideally, you are profitable on the first transaction. There are other “conversion” moments (e.g. registrations), but nothing beats acquisition and activation like real money on the table.
• Engagement: The more enthusiastic your users, the better. Use engagement and NPS surveys. Being controversial is okay, not everyone has to love you. It could be a lot worse. People could not care about you at all.

Detractors give you a much higher chance of turning them into fans. They are engaged and they care.

• Retention: Your users come back to your service and use it again with a healthy frequency. Sean Ellis wants you to learn that 40%+ of your users would be very disappointed if you disappeared overnight. This might be a bit random, but checking your retention curve is not.

Source: Coelevate

• Growth in a large market: You grow week-on-week in a market that’s “large enough.”

Each of your team members’ individual goals should directly ladder-up to at least one of those metrics. Here’s a handy list of for e-commerce business here and here.

Interestingly, I find that the sustainability of your unique value proposition (UVP/USP) is implied but not a pre-requisite to reaching the PMF. You can grow fast with profitable customers by having a marginally higher quality product if there is a hunger for the product in general. In half a year, your competition can catch up. If you don’t evolve, it can be game-over, even if you reached the PMF at some point.

4. Experiments

Look for key behaviour in data or while observing customer behaviour and design experiments around that. Test the riskiest assumptions of the hypothesis first. A few notes on designing experiments:

• Make sure you can run the test and repeatedly confirm your findings. Experiments should be designed to minimize possible errors with controls. Which parts of the product can you take away without losing customers?
• Build a central learning system for recording and sharing data from the test and learn loops. Place all experiments by all teams in one place.
• Do peer reviews. At HotelQuickly, at least one extra team member reviews any important project or experiment to challenge their design and assumptions.
• Run experiments with the readiness to change things based on results. Experiments often result in bad news. That’s ok. Commit to ongoing tests even if the initial results are failing.
• Here’s a great list of questions for customer surveys and an interesting article on conversion optimisation.

5. Team

The first and most important team philosophy you have to build at the PMF stage is that an organisation should be learning and experimenting rather than executing and optimising.

A great rule in the PMF stage for the entire team is to have them talk to 10 people a day.

Get them to use Toyota’s 5 Whys to uncover what’s really going on with your customers. What they say is never as interesting as why they said it. Second, drive team decisions with data. Myth/ego-based statements like “I feel the customer wants this” should be thrown out the window.

Scale For Growth

First things first — reach the PMF before you think of scaling. Don’t allocate resources to the growth team if your product isn’t on a clear path to sustainable user engagement and value creation.

1. Fundamentals:

Growth is 90% discipline, velocity, and cadence. The remaining 10% is in the framework. Get a grip on these four core skills; they will be the foundation from which you grow:

• Data Analysis: It’s not just about reading the data. It’s how you apply it to make decisions when building your business.
• Quantitative Modeling: Use your historical data to project possible future scenarios.
• User Psychology: Translate that data into actual consumer behaviour and emotion. Customer empathy is critical. You cannot effectively scale a product without being able to feel what your customers feel.
• Storytelling: Bring all the information together in an interesting way to engage your audience.

2. Product Channel Fit (PCF)

Once you hit the PMF, build feedback loops around your acquisition channels and cohorts.

Focus on your strengths and put your energies on the funnels that work.

Scaling those channels will yield substantially better results than trying to “fix” your mediocre performing ones. This is where you will see real growth and what will help you define your channel strategy and design the marketing mix.

Determine the metrics you are optimising for (e.g. learning, volume) and identify your constraints (e.g. time, money, target audience).

3. Experiments:

When you are experimenting for scale, you need to be able to:

• Conduct high-tempo testing: Remove the roadblocks to speed testing and promote continuous deployment. Aim for velocity.
• Prioritise: Build out a backlog of everything you think is worth testing and which key metric you expect it to move. Rate it based on effort, impact, success, and dependencies, or use ICE (impact, confidence, ease of implementation).

“Prioritisation for impact is key. Otherwise, you might get carried away by doing tons of silly quick A/B tests, like changing subject lines just to increase the number of experiments you are doing.” Kiyan Foroughi

• A note on A/B testing: Get your PMF right first. Don’t attempt A/B testing until you reach a minimum volume of statistical signification.

4. Metrics

Focus on Customer Lifetime Value (CLV = LTV), but regularly recalculate the CLV as it won’t remain fixed forever. Be particularly cautious about your churn and profitability early on.

At scale, a CLV should include almost everything, including headcount, direct marketing spend, branding spend, discounts and promotions.

Account for company overhead to get to your true gross margin LTV. Healthy LTV/CAC at scale are different for different businesses.

Source: Coelevate

Once you truly understood your customers’ journey, it can be useful to focus on specific behavioural metrics that are confirmed as indications of customer conversion. For example, Facebook’s key metric was to get a new user to invite “7 friends in 10 days.”

5. Team

It is essential that growth efforts are spread throughout the organisation, with data-driven deep cross-functional work between product, marketing, engineering, and design. Don’t run growth independently through a single functional group like marketing or product.

Once you have your team structure in place, ensure that your systems are set up for efficiency.

Hire for the right skill-sets and ensure that meetings aren’t sabotaging the speed of testing.

Distinguish between brainstorming meetings for experiment ideas and prioritisation meetings that are there to remove roadblocks, review results, and encourage action. Everyone should know what’s happening up to the hour, and check that reporting is up to date but not so granular that you waste time.

Finding The Experts

To answer my key questions and understand where HQ stands at this moment, I sought out the experts so I could connect, get feedback, and guidance on what I’ve learned so far. Research leads to a good foundation, but nothing beats interaction with seasoned specialists who have rich firsthand experience. Some experts to look up in this field that I’d recommend are:

• Globally: Brian Balfour (Coelevate), Andrew Chen (Uber), Hiten Shah (Quick Sprout, KISSmetrics, CrazyEgg), Andy Rachleff (Wealthfront), Casey Winters (Pinterest).
• In South-East Asia: Kiyan Foroughi, Kenrick Drijkoningen, You Teck, Skyscanner Growth team, Arvin Tang

Putting It Into Practice

Armed with the refreshed approach to PMF and scale, I put myself to work and went deep with the HotelQuickly team. Based on the learnings and interactions with experts, we evaluated HQ’s current state, built a benchmark with comparable companies, and defined that:

• PMF is fairly hard to define in the travel sector, especially due to the low engagement and also low repeats.
• While some players incentivise growth to drive desired behavior, ever-changing customer expectations and business models make it quite hard to sustain the PMF in the long run.
• Applying the scientific method, we outlined our fundamentals and starting experimenting. We invited experts to challenge our status quo. We reviewed our tools and usage of the central knowledge base and kept track of the biggest challenges and questions.
• There are basics we need to get right before we invest in new tools. We need to reformulate our key hypothesis and design experiments to test the riskiest assumptions fast.
• As a team, we need to rethink our approach in line with “build-measure-learn” feedback loops so that we can scale this with new opportunities. We need to get into the habit of high-tempo testing, and we need to promote a data-driven culture across our talent. Consistency in our approach (from testing to reporting and implementation) will be a key factor for us to succeed.
• We kicked off multiple initiatives to re-tool our team and got some really exciting experts on board.

Personally, as I learned and applied the test and learn approach, I realised that I need to connect with more experts to understand what I’m missing as “I don’t know what I don’t know.” I also want to dig deeper into the related sciences of statistics, psychology and psychometrics, and decision-making.

If you have any suggestions on how to better detect PMF and scale, or a great expert i can talk to, please drop me an email. Both HQ and I are committed to learn fast and grow. We are always ready to revisit our progress by taking a snapshot of where we stand, experimenting, and continuing to raise the bar.

Build-measure-learn.