A/B Testing Your App’s Appearance is Important, and it’s Free*
Build your own store testing solution in 5 easy steps!
It’s important to test your ideas early. The earlier you fail, the sooner you can change direction and hopefully find an approach that works for you.
I’m not going to go into too much background on why you should do early concept testing for new apps, or how to use dummy app store testing. If you want to get more of an overview on the subject, I recommend these articles:
The Mobile Marketer’s Guide to App Store A/B Testing by Liza Knotko for CleverTap
How to Validate Your Mobile App Idea With A/B Testing by Lina Danilchik for SplitMetrics
Why App Concept Testing is Crucial to Your App Store Optimization (ASO) Strategy, by Jordan Bennett for StoreMaven
Existing providers
The two biggest players on the market are unquestionably StoreMaven and SplitMetrics. There are other smaller actors in the field, but if you’ve heard of store A/B Testing odds are you’ve heard of one of those two.
These services are run by clever folks who have made it easy to manage assets, start tests, and will assist you in analyzing the resulting data. However, they’re also very good at charging money for all that. Last time I was dealing with one of these platforms I was quoted for $5000/month!
You could hire a couple of web developers to build an equivalent solution for that kind of money. In fact, it’d be the lion’s share of my salary and I knew that if I took this to management their first question would be “can’t we do this ourselves?”
Because that’s always the first question I get asked.
And of course we can. It’s not hard.
Cracking open the black box
As an old web dev, A/B testing features and graphics wasn’t new to me when I ended up on the store presence side. Testing is important on the web for much the same reasons as it is on the app stores, but as a web developer you get accustomed to having much more fine-grain control over the user’s experience — and complete knowledge of the user’s journey and behaviour on your site.
I knew from using StoreMaven, SplitMetrics, and other (now defunct) services that all they’re really doing is serving up web pages made to look like the app stores. They track the user behavior on the page, compile statistical reports, and show you the numbers. And you’re not in control of any of it.
Naturally, when you are running these types of tests you need to drive traffic to the test pages. This is a losing proposition since you’re effectively paying for traffic you’re throwing straight into a pachinko machine that eventually drops the users in the trash. It’s doubly wasteful because when your User Acquisition team is doing real testing they optimize on the callback events they get from attribution providers — and when doing a traditional dummy store test they have to fire blindly and hope that you have fun with the users they’re wasting.
To make this even worse, the traffic numbers often don’t match up between two sources. How Facebook reports impressions and clicks and bounce traffic is different from how StoreMaven counts them, which is different from how SplitMetrics counts them, which is different from whatever fourth source you check. This casts even more doubt on the veracity of the test, and if you want to compare the numbers to live campaigns you are shit out of luck.
And once the user lands on the page, you have no way of knowing exactly how the users are tracked, or exactly what goes into every data point. The data takes a long time to mature before it gets presented to you, and even then it can be difficult to parse.
In the end, spending time running the test, waiting on results, trying to reconcile the results with our own numbers, and then reaching actionable conclusions is just too much work for something you’re meant to pay for.
Issues in summary:
- Third-party store testing services are very expensive for what they provide
- Acquisition campaigns towards the dummy store page are nothing at all like real campaigns, and therefore the traffic is difficult to compare to live performance
- You will have issues comparing the statistics to anything besides the internal performance of the one A/B test you’re doing on the one platform you’ve bought into
- Analytics are slow, and you have no way of knowing if what you are getting are the live stats or post-processing
What if you could do a test like this yourself, for free, with raw analytics in real time, and live callbacks to the campaigns that drive traffic to the test?
I can make that happen for you. You’re gonna need a little bit of basic web development mojo, but if you’re making an app I’m sure you also know your way around basic HTML, CSS, and Javascript. If you’re not a developer yourself I guarantee that there is someone within Nerf Gun range that can help you out. Or you can hire someone to do a day of work.
The first time I set this thing up it took me all of 3 hours. But honestly, even spending a week setting this up will be worth it.
Build your own store testing solution in 5 easy steps
1. The Dummy
The first thing you’re going to need for this is a dummy store page — something that looks enough like the App Store or the Google Play Store to get indicative behavior out of the users you send to it. Of course matching them exactly might get you in trouble if anyone were to complain, so I recommend landing somewhere in the “plausible deniability”-valley. It’s also for this same reason I’m not going to share my own templates here. I’ll just say you can probably find some HTML templates out there — especially if you’ve used a third-party testing solution lately.
2. The Analytics
You’re gonna need a basic Google Analytics set-up, as you’d use for any old website. When you have that set up and wired into your store dummy page’s code, you can put analytics triggers on anything and everything you want to track! User taps download? Send an event to analytics. User expands description? Send an event to analytics. User interacts with the screenshots? You best believe that’s an analytics event! You don’t need any fancy code for this, just an onclick trigger that sends a single event name to Analytics and you’re good to go.
3. Hosting
Yeah, you’re gonna need to put the dummy somewhere. Personally I just drop everything in our Google Cloud Storage but it works just as fine running from any old web hosting solution. Run a server on your own machine if you want!
With just these three things, you could start running traffic to the dummy and get user behavior. And guess what? Google Analytics reports in real time! But you won’t get A/B variants, so let’s get that set up.
4. The Variant Testing
Remember when I told you that A/B testing on the web wasn’t a new thing? Yeah, Google had A/B testing built into Google Analytics and Webmaster Tools years ago — for free! Since then they’ve discontinued that service… in favor of spinning it off into its own thing!
Welcome to Google Optimize — your new best friend. Setting this up is probably going to be the hardest part of this, and it’s not especially difficult. You’re going to need a Google Analytics Account (which you already have from the previous step!), and some settings toggled in Google Tag Manager, but Optimize has really good guides for setting all of that up.
Optimize is free for running up to five simultaneous tests. Above that and it starts to get really rather expensive, but five simultaneous tests should be enough for any company that thinks StoreMaven or SplitMetrics is expensive.
Once you have Optimize set up, you can test your dummy by setting up a new test and just feeding it the URL to where you put your dummy page. Optimize will run a diagnostic to make sure you set up the tracking correctly. If not, do some troubleshooting, Google some errors, read some StackOverflow — you know, the usual.
When Optimize successfully validates your tracking set-up, you’re good to go! You can run your first test! Here is where the magic is: The implementation you just did? It allows Optimize to tweak the content of the page for your visitors. This means that you can use Optimize to create variants on your store dummy without changing the files stored on your host at all! Google will automatically divide the traffic coming to the page into those seeing the unmodified dummy, and those seeing your variant(s), and track their metrics separately.
All you need now is to set a main conversion goal to track and send traffic to the page. Optimize will give you your first statistical reports within hours.
If you are accustomed to running store asset A/B tests on Google Play, you will be familiar with the kind of reports you will get out of Google Optimize, but these are even better. You will get a history of the conversion rates split between the variants, with confidence intervals and all the usual good bits. Of course you can also go into Analytics yourself and check the performance of all the various events you decided to set up as well. It might not be quite as pretty as the type of reports you would get out of those third-party providers, but it’s yours.
5. UA Callbacks
I promised you wouldn’t have to waste your precious acquired traffic earlier, and don’t worry — I intend to deliver on that (as long as you like Facebook!)
What you just set up is just a web page, and any performance marketers you have in your vicinity will tell you that doing performance-based user acquisition is totally viable on the web — you just need a Facebook Pixel. They’re really quick and easy to set up, just add Facebook’s tracking into your store dummy’s HTML, link it to your advertiser account, and set up the events to fire on the things you want to track — just like with the Google Analytics you set up earlier.
This is going to mean that when you run ads on Facebook to send people to your dummy page, and those people tap to download — you will immediately see that in Facebook’s dashboard. Again, in real-time. This will make it easy for your UA managers to get an immediate overview on how the test is performing, by tracking the Lead Event you are sending them from the page.
Now, a disclaimer: You won’t really be able to optimize for these events exactly as you would install events for a live app. It usually takes too long for Facebook’s algorithms to learn and optimize on new events like this to actually get a feedback loop going on a week or two of testing. However, this type of set-up will allow you to see the conversion events in Facebook’s statistics for the campaign you’re running, which is more than you’d get without it.
So, are these numbers any better than the third-party reports?
I’m sure you will have your own opinion on that, but I can tell you that on our end, what we’ve seen that live apps that have come out of this testing pipeline have remarkably similar install rates as their dummy stores did in these tests. With previous third-party solutions we often had significant differences.
And personally, I am infinitely more willing to trust numbers from Google Analytics than numbers from third-party store testing providers. Because it means I have much fewer blind spots, much fewer niggling issues of comparing discrepancies, and much fewer other parties I can blame for strange numbers.
That all adds up to running tests for a bargain, that I am more willing to trust to give recommendations and take action on.
Conclusion & Shout-Outs
I hope you found this post helpful! If you’ve been feeling like these types of tools should be possible to set up in-house, I hope you leave reading this feeling vindicated and excited to get to work on it.
I want to give a shout-out to Danny Armstrong at PlaySide Studios for helping out with some of the early work on this project. Also kudos to our friends at Hutch Games for their feedback early in developing this. Another shout-out to the folks behind ASO Giraffe who had much the same idea as me at the same time, and built an affordable self-service platform along with it.
