Product Experimentation During the COVID-19 Outbreak — My Weekend Thoughts
The COVID-19 pandemic and its wide, global implications caught all of us by surprise, leaving us in the face of uncertainty. The pandemic has ramifications on the personal life and professional decisions alike.
A few days ago I talked with a colleague about the implications of the COVID-19 pandemic on our product experimentation and discussed what adaptations, if any, are in order. I’ve thought about the issues of product experimentation some more — and here is my personal take. I assume that similar questions cross the minds of many product professionals — hope you can find some answers here.
Continue running product experimentations
In general, product teams should continue running experiments. There are exceptions and balances that have to be taken per industry and geography (e.g., the travel industry has already suffered a major impact) — but my view is that experimentation is still valuable.
What experiments to run and how?
While I think that experimentations have to continue, here are a few guidelines:
Reevaluate your hypotheses — make sure they are still valid. Have a second look at the expected results to assess the validity and extent of the expected impact. This is particularly crucial in case the hypothesis is data backed (rather than pure learning), as there is a high probability that the past data doesn’t adequately reflect the current users’ behavior.
Prefer short term experiments over ones that are part of a long-term product strategy — there’s a good chance that the results of the current period can’t be averaged into the stable past results (and maybe not into the future results).
Make sure you can flag the experiments run during this period so you’ll be able to exclude them, if necessary, from the hypothesis creation analysis and of future experiments.
Stick to methodological and proven testing practices, and consider the following:
- Keeping a (small) treatment group after the experiment has been completed, to be able to validate the results over a longer period and adjust if and as needed.
- Repeat (some) of the experiments in the future for revalidation.
- Keep the experiments running for a longer period to reach a higher level of certainty.
- Be even more suspicious of extreme results.
If your product is part of a longer user journey (for example, marketplaces where the user passes from the marketplace to the service provider’s experience) — review the experience on the provider’s end and evaluate or tune your experiments accordingly.
What to avoid and why?
Experiments that involve users’ long term commitment, such as annual subscriptions. Not only are people less comfortable making such commitments, experimenting with longer-term offerings implies obliviousness of or indifference to people’s economic stress that is so widespread these days.
Price related experiments — people are much more price-sensitive, so their actions and choices are not indicative of their normal behavior. In addition, prices are likely to be volatile which may affect the experiments’ results.
Device (mobile/desktop) specific experiments — people are spending much more time at home due to quarantines, skewing normal usage patterns of mobile and desktop.
Experiments that involve significant changes — especially ones that are heavily dependent on infrastructures that are highly affected by the COVID-19. For example shipping, delivery, non-online support.
NPS / customer satisfaction experiments — except in specific cases or industries, these are less likely to reflect normal times: people are either eager to reach immediate results (e.g., health-related) or have way less attention for such requests.
Final note
During such a global crisis that affects all of humanity, sensitivity to your customers’ needs is even more important than usual. While all of us want to reach successful results, we are required to an even greater level of responsibility and transparency. Keeping your customers updated about expected hurdles in your inventory, shipping, availability of services across locations and more, is not only expected in such times but will also be appreciated by your customers long after the pandemic will become history. Keep this in mind while experimenting.
