Is A/B Testing Worth It for Bootstrappers?

“Companies that use innovative and data-driven analytical approaches to marketing are found to have the highest success rate of conversions on their website.” — WeSpotlight

In case you’re not familiar with A/B testing, here’s a quick definition from Visual Website Optimizer: “A/B testing (sometimes called split testing) is comparing two versions of a web page to see which one performs better. You compare two web pages by showing the two variants (let’s call them A and B) to similar visitors at the same time.” Then you record how those website visitors behaved differently based on which version they were shown.

After analyzing the data, you keep whichever design performed better, discard the other one, and start the process again with a new tweak. Over time, you iterate toward the Holy Grail: a perfect landing page that converts 100%! Jkjk, that’s impossible — but you can certainly improve your baseline. A/B testing is a simple way to ensure that the changes you make are doing what they’re supposed to.

A/B testing is not a magic wand. Illustration by Alexander Baxevanis.
A/B testing is not a magic wand. Illustration by Alexander Baxevanis.

This is obviously a very clever idea. Equally obviously it takes time and energy to pull it off. If you have limited resources and you’re forced to be ruthless about where you focus your effort, is it worth it to A/B test? You have to come up with website variations, deploy your A/B testing tool(s), and then wait for enough visitors to be processed. The concept is simple, but the execution is often lacking. From the Kissmetrics blog:

“A/B tests are designed to imitate scientific experiments, but most marketers running A/B Tests do not live in a world that is anything like a lab at a university. The stumbling point is that people running A/B tests are supposed to wait and not peak at the results until the test is done, but many marketers won’t do that.”

As a bootstrapping entrepreneur, maybe with a day job to balance, it’s important to evaluate where your energy will have the highest impact. Of course, “is it worth it to A/B test?” is one of those trick questions: the real answer is that you have to weigh your priorities and decide for yourself.

This issue is on my mind right now because I’m personally debating whether I can justify spending time and effort A/B testing the Product Communication Basics landing page. And I think the answer is… no. Here’s why: opportunity cost.

Investopedia handily defines opportunity cost as “the cost of an alternative that must be forgone in order to pursue a certain action. Put another way, the benefits you could have received by taking an alternative action.” Right now I don’t even have enough website traffic to get statistically reliable results from an A/B test! So I’m gonna work on that first. Hi Reddit ;)


Do you need a practical guide to writing sweet landing page copy? Check out Product Communication Basics.

If you enjoyed this article, please subscribe to be notified when I write a new post or release a new product.

Originally published on Product Comms Club.