How A/B Testing Got Us 100,000 Users In One Day
Much of this information is taken from my new book Explosive Growth, which you can purchase on Amazon.
In 2007, my company was in a precarious spot:
- We’d scrapped our product for a full rebuild.
- We were rebuilding our product as a Facebook app — a totally untested platform.
- We were quickly running out of funds.
We needed our new dating service, AreYouInterested?, to find traction quickly, otherwise we’d be out of capital.
I’m going to go ahead and spoil the ending — It worked out. AreYouInterested? is now FirstMet, a dating community with over 30 million users.
However, back in 2007, we had no idea it would work. We were sure of just two things:
- Facebook had a giant, built-in user base.
- Big companies had yet to tap into that user base.
If we could figure out how to make our Facebook app “go viral,” we could tap into explosive growth before any company with a bigger budget moved in to compete.
#ExplosiveGrowthTip: Many of today’s billion-dollar companies succeeded by growing on top of other platforms. Are you testing integrations with emerging and established platforms? Visit the Explosive Growth website to see more explosive growth tips for entrepreneurs.
The problem was, virality wasn’t as well-understood then as it is now. There was no handbook for going viral on Facebook — there were barely even Facebook apps at this point.
And so, we did what any startup full of under-informed but hyper-motivated people would do, we didn’t sleep.
How A/B Testing Made Us Virality Experts
Before you launch any test, you need to know what you’re testing for. In other words, every growth experiment must have one single result that defines its success — in marketing parlance, we call this a KPI (Key Performance Indicator).
The idea is that “growth” is too abstract a thing to be your only KPI. What you need is a concrete result that leads to growth to test and measure.
For us, the result we wanted was for Facebook users to invite their friends to join our app. Friend invites were our KPI, because we knew that was the one user behavior that would lead to viral growth.
Now, at this point we knew the following:
- We couldn’t grow unless our users invited their friends.
- Our users needed an incentive to invite their friends.
- The friends needed a reason to accept the invite and try the service
#ExplosiveGrowthTip: Can you identify the one thing about your product that users want more of? Have you tested offering this to them for free if they get some friends to join?
We didn’t know what that incentive should be or how we should offer it, and this is where the sleep deprivation comes in.
We Spent 18 Hours A Day A/B Testing
In order to learn what worked and what didn’t — remember, no one was in expert in Facebook virality yet — we had to try a lot of different things. I mean, a lot.
We were operating off of one assumption:
On a dating app, users (especially men) are generally looking to get more incoming messages. So, giving users an opportunity to get more attention (thus more possible messages) would be the perfect incentive.
To provide the necessary reward system, we challenged our users to invite five new friends, and as a reward for their efforts, they would appear higher in search results, which would lead to more matches and messages. Eureka! Almost every user invited five of their friends.
After that, we thought that a good idea would be to up the ante, so we increased the challenge to ten friends, and almost every user accepted that challenge as well.
The maximum number of friends that Facebook allowed us to ask for was twenty, and that number gave us thousands of new users every day on the app.
We also realized that small changes in language could have drastic effects on the results. The following is an example of how we changed the language ever so slightly to tap into a user’s emotions, and get them to act:
- Iteration one: Invite your friends!
- Iteration two: Invite five friends for higher placement in search results!
- Iteration three: Invite twenty friends for higher placement in search results!
- Iteration four: Invite five friends for more matches!
- Iteration five: Invite twenty friends for more matches!
- Iteration six: Invite five friends to find out which of your friends likes you!
- Iteration seven: Invite twenty friends to find out which of your friends likes you!
By changing a few words around, we got drastically better results.
One key item we learned in this process was that “selling the benefit” was far more effective than selling the feature.
The feature is what something is, and the benefit is how it improves lives. For example, people weren’t enamored enough with the “feature” of getting higher placement in search results, because it wasn’t clear to them why this was important.
But getting more matches, or finding out which of their friends liked them was an emotional benefit they could associate with some value. Those small changes were a big part of how we got to over a hundred thousand new users per day.
A great example of selling the benefit and not the feature is when Steve Jobs introduced the original iPod. The feature was a five-gigabyte hard drive, but the benefit (and the slogan they used) was “one thousand songs in your pocket.”
#ExplosiveGrowthTip: When writing copy, sell the benefit, not the feature. Are you selling the benefit?
Want Explosive Growth? Test Everything
We saw the slightest change in copy and design (including something as innocuous as a color change) had a 100 percent or more improvement on the results.
Believing that our success hinged upon our ability to learn quicker than our competitors, we built an internal testing platform that enabled us to run hundreds of simultaneous experiments.
#ExplosiveGrowthTip: If you’re not running tests, start now. Do you plan to run at least three tests in the next thirty days?
Rigorous testing is something that any business — even something as simple as a corner store — could benefit from. Test what the sign in the window says. Test product placement. Test what’s in the display case. Test pricing.
Even a bunch of small increases taken together might equal a large increase.
Two books on experimentation and a/b testing that I highly recommend are: You Should Test That by Chris Goward and The Lean Startup by Eric Ries. Here is my list of the best books for entrepreneurs that helped me succeed.
If you enjoyed this, please clap below and share with friends.
Cliff Lerner is the author of the #1 Best Seller Business Book, Explosive Growth — A Few Things I Learned Growing To 100 Million Users. For more Explosive Growth insights, check out my PR Tips To Get Free Press, Viral Marketing Playbook, and Top 10 Entrepreneur Growth Tips.
This story is published in The Startup, where 263,100+ people come together to read Medium’s leading stories on entrepreneurship.
Subscribe to receive our top stories here.
