Hello, ladies and gentlemen.
We are glad to share another portion of news how to run experiments properly. Our study is based on the work with Amsterdam escape room Questomatica (questomatica.nl). It was founded by a charming woman Zina back in 2015.
Beginning
Like we always do at the beginning it’s all about understanding the current analytics to enhance its state further. The case of questomatica.nl is unique for several reasons. Firstly, due to historical reasons, it has two different domains: questomatica.nl for the landing page and listing of the quests and book.questomatica.nl for the booking page. Secondly, due to unavoidable circumstances, they use the Events to track the bookings instead of e-commerce analytics from Google Analytics. Well, it sounds like an interesting challenge!
The new events we added to Google Analytics:
- Depth of the scroll
- Views of each section on the homepage
- Clicks to links in the menu, “Make a gift” link and “Begin adventure” links
Test Zero: A/A to prove the installation is correct
Two weeks after the launch we did not trace any significant changes and hit the ground running.
Test One: Change the visual hierarchy of elements in quest block
Hypothesis: the current layout makes more stress on the information about age/participants restrictions. Even though we admit that’s necessary, we wanted to try to move focus to the Book button to get more clicks on the button and therefore more reservations.
Change: make the button brighter and wider and remove the background from “restrictions” element
Result: this experiment has not worked as expected
Test Two: Change the picture of the quest
Hypothesis: people in the picture can attract more customers
Change: Zina kindly provided us with different photos for the quest, and we found one which seemed us very suitable
Result: off the mark
Test Three: Pricing section on the homepage
Hypothesis: from our experience for Dutch people the price is essential. Therefore the existence of this section on the home page can be appealing to potential customers because they can see it’s not different from other escape rooms in Amsterdam.
Change: we added to the home page simplified block with a price
Result: the number of transactions increased but not so significantly as we expected so we could conclude it was because of the change and not just a statistical error.
Taking a step back
At this moment of time, we started to be worried — the A/A experiment we launched in the very beginning showed the significant difference between groups that have no changes. That means one simple thing — we cannot trust our results.
First of all, we checked the technical side of our system. Everything worked as expected — the transactions happened when they should, no flaws on the experiment runner side, both Control and Experiment groups are assigned correctly to the users.
Then we dived deep down in Google Analytics. Thanks to the way Growity integrated with Google Analytics we can retrospectively that all available data there and not what we tracked at the start of an experiment.
In total, I guess we spent near 30 hours of real working time to find out the reason.
The difference in the conversion rate between groups was created by “artificial” user(s) who visited the website more the eight hundred (800!) times and did not purchase anything. We assume this is a Google Bot, Bing Bot or something similar. In fact, excluding them restored the difference in the number of sessions for groups, which in its way restored the Conversion Rate and did not change the number of transactions.
Test Four: Fun badges
Hypothesis: after a long discussion with analysts about the elements that can be added to the website we decided to try something unusual. Taking into consideration that many users are novice, experiments were conducted in summer, and there are a lot of stag-parties in Amsterdam, we decided to give a hint what quest is suitable for visitors. Wiki says that the average age of marriage for men in the Netherlands is 33.5. That means that the childhood of those people was in 90th — 00th. Well, that’s good, because one of the quests of Questomatica is based on 90th. Thus, the badge about 90th seemed to be a great addition.
Change:
Result: now, when we were confident that our system worked correctly we concluded, that with the confidence level of 84% (statistically significant) the badge improved the conversion rate.
Test Five: Booking progress
Hypothesis: at that moment we checked the popular CRO changes to the websites. One of them we found fascinating — to show the stepper on the page with the timetable.
Change:
Result: we achieved statistically significant 29.42% difference in conversion rate with the confidence level of 90%.
Wrapping up
Growity team expresses gratitude to Questomatica and Zina for the cooperation. We were approving all changes and designs by here, and that helped us to achieve our goal!
