“Let’s A/B it.” Sounds so reasonable, doesn’t it? Your team is split on two directions for a feature, and both have merits. So why not A/B test it?
I mean, think of the benefits:
- You get cold, hard numbers to backup your decision.
- Customers tell you what’s better through actions, rather than opinion.
- Your team’s feelings don’t get hurt since both approaches get tested.
- You’re being methodical and scientific.
- Tools like Optimizely make it easier than ever before.
So why not do it?
Because your startup doesn’t have time.
If you’re a mature company, churning out cash like Google or Facebook, you bet you’re running A/B tests. You’re deploying to millions of people, and even a 1% increase can be worth a boatload of money. You also don’t want to hurt your current business, so you’re going to A/B the crap out of it.
In a startup though, too much A/B testing will kill you.
While you’re optimizing signup conversions, your competitors are building the next big thing. They’re adding a referral program, creating a sea of SEO content, and instituting new features that will blow your customers’ minds. 5% lift in signup conversion? Psh. You need a 5X lift in customers trying your product. Every month. For years.
And that means you don’t have time to make both Feature A and Feature B. Or I should say, “Features A, B, and C,” since you also need Feature C—the one that runs the experiment, divides populations, and collects results.
The complexity adds up. Instead maintaining a single Feature A, you now test and debug Feature A, B, and C, as well as the interactions between them. This gets gnarly fast, and the costs rise nonlinearly every time you add a new scenario.
Instead, keep it simple. Ship Feature A when it’s done. You made Feature A because it’s valuable, so every moment it waits for Feature B or C, you lose money. Once A is out, you can learn from customers’ reactions and iterate—if that’s really the best use of engineering time. Usually, Feature A is good enough for now. Especially since your product has areas with no A or B yet.
Too often, A/B tests are a crutch — a way to avoid hard choices and make decisions feel risk-free. But it’s not risk-free. The costs are very real, and like a crutch, A/B tests slow you down. As a startup, you just can’t afford that.
