4 tips when running Lean Startup experiments during COVID-19
The COVID-19 virus certainly shook things up. We’re living in the new normal. Every person, company and government is impacted in some way. Some have seen positive changes, while others are in survival mode. This certainly affects entrepreneurs and intrapreneurs trying to validate their idea or business model.
I’m coaching several startup teams and I rounded up my lessons from the months on how you can keep learning and validating with the Lean Startup methodology during these strange times.
The past does not always predict the future
The Lean Startup from Eric Ries and Customer Development from Steve Blank are based on the notion that the past is a good predictor of future behavior. For example, during customer interviews we ask for specifics in the past instead of generics or opinions about the future. Only the first can be used to as valid data points to (in)validate hypotheses.
So, the past is essential when validating your business model. But now there is COVID-19. And no one knows what the new normal will bring us and how this will impact the future.
Starting businesses are struggling with the following question: “Can we still learn from talking to customers, what information can we use to build a business on?” Here’s what I think you can do to keep learning and make sure you’re not building a business based on validated assumptions that are suddenly not valid anymore. TLDR;
- Re-evaluate your completed experiments and lessons learned, things might have changed.
- During interviews, look for clues that indicate permanent change.
- Offline interviews can be done online through Zoom, Hangouts, Skype, etc. but make sure you can see facial expressions.
- Online behavior has changed, higher traffic but less conversion.
#1 Re-evaluate your past experiments and learnings
Let’s first look at the experiments you already ran. Can you still trust the results? My advice is to re-evaluate the experiments you already completed.
The lessons learned from those experiments might have become useless due to government regulations, company rules and shifts in priorities.
If your customer segment told you that traveling was a big part of their day, then that’s probably not true anymore. And if someone told you that face-2-face meetings were required to get things done, ask again today and you’ll get a different answer in most cases. In addition, priorities might have changed
For each of the learnings that are impacted, ask yourself the following question: Is this change temporary or permanent? If you’re not sure, go back and talk to your customer. Read on to learn what questions to ask.
#2 Finding out if a change is permanent or not
Your customer might not know what the future holds, and you can’t trust their predictions. So to learn if certain behavior will stick after COVID-19 you have to take an indirect approach. Look for clues that indicate permanent changes.
For example, you are interested in knowing if people will keep working from home, ask about investments, commitments etc. they or their employer has made. A friend told me his employer gave him a budget of €1500 (!!) to upgrade his home office, allowing him to buy a large LCD, a desk and a desk chair. Companies don’t make these kinds of investments if they think the ‘remote work’ situation is temporary.
Another B2B example is: “Do you use SaaS software for online collaboration? Does the company pay per month or per year?”. A yearly subscription is obviously indicating a more permanent change.
For B2C interviews you can do the same of course. We’ve all seen gyms taken their exercise classes online using livestreams, so that gym members can do workouts from home. Question to members: “When did you do a workout from home? How did you find out about the workout? Did you buy any gear to have some variety in workouts?”. From these questions you can learn if the interviewee is looking for workouts himself, and if he/she bought gear, this indicates they might be doing workouts from home when gyms open their doors again.
#3 Running new experiments — offline
Offline experiments are interviews, focus groups, surveys, contextual inquiry, observations etc.. These offline experiment types often imply direct interaction between you and the person you’re interested in. Most of these are crippled now because of government regulations which discourage direct contact with others (or keeping people inside of their homes).
Despite this situation, teams at Aimforthemoon have been running (B2C) interviews in the streets of Amsterdam. One of the key findings was that people were really happy to talk to someone and didn’t mind to answer some questions (as long as they kept distance). In some cases interviews took over 20 minutes.
B2B interviews might be a bit harder to do in person. However, most businesses are now used to do have video calls every day so this also opens up more possibilities. I agree with Steve Blank that these experiments can be done through Zoom, Hangouts, Skype etc.
Make sure you can see facial expressions while doing your thing, as this often holds valuable information.
By the way, need help with customer development interviews and setting up experiments? Check out my post “ A curated list of customer development resources (40+) 📖”. It’s a one-stop shop with the best reads on custdev I found online.
#4 Running new experiments — online
Internet and social media use soars now that lots of people are working from home. Startups, SaaS startups in particular, see opportunities in this because the group of potential customers to reach out to has grown. More online traffic lowers ad costs because there’s less competition between advertisers (competition drives up the price per click). When the total amount of money companies put in advertising stays the same, it should result in higher conversion numbers.
At least, that used to be the case. Currently, several sources report declining conversion rates through online advertising. Google Adwords showed a 21% drop in conversion on average (this differs by industry). The upside is that the cost per click is lower.
Facebook and Instagram show lower conversion rates and lower CPM (cost per thousand views) as well.
Why bother? Well, the drop in conversion might skew your experiment results, you should expect less conversions, and higher Customer-Acquisition-Costs (CAC).
An example: your team is using Google Adwords to draw visitors to a landing page. On average there’s 20% less conversion using these ads. Your hypothesis is that 10% of the visitors will convert and leave their contact details. The “10% conversion” is what we call the success criteria of the experiment. This percentage is defined before conducting the experiment. Currently, you should aim for 20% less conversion, so the success criteria should be 8% instead of 10%.
Lower conversion means you have to spend more on acquiring customers (Customer Acquisition Cost metric). My advice is to investigate what the new status quo is for the sector you’re in. There’s quite some research and numbers available on this.
Some final words: inspiration of great businesses started during a crisis
Uncertainty sucks, for established and beginning businesses, for B2B and B2C segments. Everyone is impacted. Moreover, consumer and business confidence are falling, stock prices are declining. You might get discouraged by the current situation. But keep in mind, in every situation there will be winners. To remind you, here are some articles about iconic companies that got founded during crisis and recessions: 1 and 2. Let’s go!
Originally published at Linkedin.
