Alex Chuang,
Keevin O'Rourke

Hi Keevin, thanks for the kind words and appreciate the response. I whole heartedly agree with using scenario B. One of the challenges that entrepreneurs have faced is that the sample size they are testing the experiment with is just not statistically significant enough. Therefore, it’s typically hard to validate an experiment when the sample size does not make statistical inferences about the population. That being said, the other challenge I’ve seen is the nailing down who your early adopters are. Entrepreneurs who are building consumer apps typically target everyone. While they may be able to get a decent sample size, they will struggle with gaining traction early-on due to lack of focus.

This is why at the early-stage, I think a lot of it do rely on your gut instinct and good UX practice. Until you are able to get a decent sample size that well represents the early adopters you are targeting, tests like A/B testing and variable testing are meaningless because the results you get could still have large margin of error. After all, startup experiments are conducted to hopefully decrease your uncertainty about a hypothesis. Whether you will yield greater precision or not is entirely dependent on the number of variables you used, size of your sample, characteristics of your sample and the overall design of the experiment.

P.S. Time is important. Getting a larger sample size takes time. There’s a trade off. Entrepreneurs must decide whether the experiment is worth it or not. If you test every little thing (e.g. color of your button) at the early stage, you will most certainly fail.