Beta testing & beta tester acquisition through Snapchat

A few weeks ago, we had the chance to talk to Scott Belsky during the botcamp program at betaworks. He talked about what he calls the first mile experience of a product.

It made us realize how important it was to focus as much on the initial experience of new users than on building product around behaviors of power users. However, we quickly noticed how hard it was to take a good metrics-driven approach to it:

• Being in beta usually means lacking established channels of acquisition. It prevents you from driving significant cohorts of new users and get relevant metrics.
• The burden of distribution for beta apps (Testflight, if you hear me!) alienates normal users limiting your tester audience to tech early adopters.
• The infinite curiosity of friends, investors and everyone asking you to try your product significantly skews your early data.

In order to avoid these pitfalls, we implemented the following:

1/ We worked on a relatively time-consuming but easily repeatable way to drive relevant new cohorts of users, especially for our beta-testing. For us, since we’re targeting US teens, we chose Snapchat as a user acquisition channel. We find new users (for example from Jake1 to Jake200), add them as friends on our company account and then send them a snap promoting our app as you can see below:

2/ We changed the name of the app, removed all the keywords & meta and published it to the app store to remove any app store discovery and make sure the only way to download the app is through Snapchat.

3/ We started saying no to friends, investors, partners, or everyone we know who wanted to try our app to leave our cohorts unbiased. When we can’t resist, we make them try on specific accounts that we can easily set apart in our analytics.

4/ We decided not to tell our new cohorts of users that our product is in beta to foster “real usage” versus “testing”. It completely changed how users used our product. People who wanted to break our conversational AI by asking random questions were now just using it like a normal product.

This approach provides us an easy way to iterate on how to market the app by changing the snap we use to attract new cohorts of users. We analyze the “snap conversion rate” which is the ratio of people who download the app compared to the number of people who actually saw the snap.

It enables us, for every new release, to drive unbiased new cohorts of users and collect very relevant data so that we can decide on the next step and iteration.

Build, beta-test, rinse & repeat!

Of course, I’d be happy to know if you have feedback on our approach or if you found other ways to correctly beta-test your product.

🤗 🤗 🤗