Kicking the tires on A/B testing
We just concluded our first A/B test on Firefox Send and we’d like to share our results. This being our first test we wanted to start with something super simple just to get comfortable with the tools and reporting. To be totally honest the change we ended up making was so minor, in my mind, that I didn’t expect to learn anything interesting and fully expected both the control group and experiment group to show nearly identical behavior. Annoyingly, I did end up learning a few things.
The Experiment
We wanted to see if we could increase our upload rate with a tweak to the UI. Here’s what the original, control group, page looks like…
We made a slight change to two styles on the main page. We increased the thickness of the dotted line border around the file drop area and made the file selection button slightly bigger with bolder text. The hypothesis was that the increased “beefiness” of these elements would make it easier for folks to notice what to do next, thereby increasing the rate they successfully upload a file versus just leaving the page. Here’s the experiment UI…
We use Google Analytics (GA) to run our experiments and to collect anonymous usage statistics to help guide us in improving the service so I want to take a moment to explain how we try to use it respectfully. We keep a list of all the types of events and data we gather, limiting what we collect to things directly helpful to improving the service. If you’re opposed to sharing this data with us the easiest way to opt-out is to use Send in Private Browsing Mode. We also respect the Do Not Track (DNT) browser setting. Consequently, doing so will also exclude you from experiments like this one. We care about making our products better but we also care about your privacy.
We launched this experiment on September 13th, but due to a configuration snafu on my part we didn’t start getting good data until the 14th. This was one of the minor annoyances with setting up, and getting used to, GA’s version of A/B experiments. It takes about a day to see any new data so it took a day to notice the mistake and correct it. Once the experiment was running smoothly the rest was mostly just a waiting game to see how each variant performed.
The Results
Early results showed the Beefy variant outperforming the control by a fairly significant margin but by the end they were performing pretty close to the same in terms of visit to upload conversion rate. The lesson here is to not declare a winner too soon.
Overall the Beefy UI had a conversion rate of 37.47% compared to the original at 36.18%, a 3.58% increase. Not mind-blowing but respectable.
In addition we used Data Studio to analyze some other behavior. The most interesting to me was the method of upload, being either drag & drop or a click on the button.
In the first few days of the experiment Beefy had a 14% increase in drag & drop compared to the control. Over time, as with the upload conversion rate, it started to even out, but it maintained about a 4% increase at the end. From a conversion standpoint it doesn’t matter how users are choosing to upload, but it is interesting that the change in the border thickness that I thought was barely noticeable even side-by-side, actually had a measurable effect. Small changes can make a difference.
Conclusion
Since Beefy performed slightly better we’ll be using it as the default UI in our next release. We learned a bit about our tools, our workflow, and our users. We plan on doing some more, and more exciting, experiments in the near future and are excited to see what we learn and to share our progress with y’all.
