Google Optimize Sunset: Comparing 3 Free/Cheap Alternatives for Shopify A/B Testing

Google is sunsetting Google Optimize later this year. I've been digging into alternative tools to consider if you want to continue running A/B tests on your Shopify store, without breaking the budget.

Introduction

There are a lot of A/B testing tools, but Google Optimize has been the choice of many Shopify stores over the years for a few reasons:

• free to use
• easy to set up and integrate with Shopify
• simple to run quick A/B tests using the Visual Editor
• powerful features for more complex experiments
• automatic reporting of results powered by GA data

I'll evaluate alternative tools in this article, using those points as the criteria.

If you're reading this, you're probably already running A/B tests on a Shopify store, or considering starting. If that's you, let's find the best tool for you! Maybe you also want to join our experiment club — each week we choose an A/B test that you can run on your store, and we send you a simple step-by-step guide for how to set it up.

Tools Considered and Pricing

I've selected three tools for consideration, and here is their entry-level pricing (based on 50k monthly store visitors).

• VWO — Free
• Convert — $99/month
• GrowthBook — Free (up to 3 users)

Other interesting options without a free trial or entry-level pricing are Optimizely, ABTasty, and Kameleoon. I may review these in a future post.

From a pricing perspective, VWO is free and therefore a good option for around the 50k traffic volume. However, once your traffic is 100k, Convert is cheaper. GrowthBook scales the best from a pricing perspective for small teams, as their pricing is based on users rather than traffic.

Wisdom of the Crowd

Before looking at the tools, we can harness the wisdom of the crowd. Let's start by taking a look at what tools people are already using:

VWO has around 3x as many users as Convert, and GrowthBook is nowhere to be seen. Although GrowthBook is a relatively new player, I do think the stats are a good indicator of which tools to look at. Notably, Optimizely also has high usage. Please give us a free trial Optimizely!

As another point of reference, here is the G2 grid for A/B testing tools:

VWO comes out well again in the coveted top right section, while Convert also has reasonable Satisfaction, albeit with a smaller Market Presence.

Setup & Integration

I set up each of the three tools for my demo Shopify store CurvyTie.

Here's how it went with each tool:

VWO

Very simple to set up — copying their "smartcode" snippet into my Shopify theme was enough. VWO also has a Shopify integration, which might be an even more straightforward way to get started, but I didn't discover this until after they'd sent me down the "smartcode" route.

Convert

A very similar setup process, just copying a code snippet. The instructions were clear and straightforward.

GrowthBook

It was challenging to set up. Even though I have a technical background, there are several ways to integrate the GrowthBook code, and no clear guidance as to which would work best for Shopify. I did get it working in the end, but I would only recommend the tool if you have Javascript knowledge.

The Visual Editors

The visual editor in Google Optimize was our go-to feature for creating quick and simple tests. Let's take a look at the visual editors in our new tools.

VWO

The VWO visual editor is great! It has a nice UI that surfaces the most important features.

I could easily make simple variant changes: editing text, changing styling, and hiding an element. I found their "rearrange" feature clunky though, which is something I would want to use a lot.

The editor has several ways to edit your page: inline, edit element, edit HTML, and add code. Each of these is useful in its own way.

The "select relative element" feature also works well — it's easy to browse the element tree, which we often do when creating variants.

Finally worth a mention is the "Library", which gives you a load of commonly used elements that you might want to add to your page, including simple HTML elements like lists and buttons, as well as more complex ones like modals and popups. Nice!

Convert

The Convert visual editor is still in beta, but it's pretty good for a beta!

The UI is simple, and making visual changes is quite easy, albeit the options are limited compared to VWO. Once again, the "rearrange" feature is clunky — maybe I'm just not good at that!

However, I'm missing some of the excellent features from VWO — such as browsing the element tree, and adding elements.

GrowthBook

When I tried it out, the visual editor was still in beta, and it didn't work with Shopify (probably an iFrame issue). Their support acknowledged this. There is now a new version, but it's restricted to Premium plans. At this point, I'm ready to rule out GrowthBook as the tool for us.

Experiment Settings & Configuration

Both VWO and Convert allow you to define your experiment's scope by combining page and audience filters.

The Convert template audiences seem particularly well-considered for the tests we will likely run.

I feel both miss the complexity of configuration we are used to from Google Optimize though — for example when defining variants across multiple pages.

Metrics & Reporting

Both tools have similar capabilities when defining goals for experiments. VWO allows you to create a definition based on events and properties, while Convert again uses a template-driven approach.

Either way, both tools have what we need to define our typical goals — click-through rate, bounce rate, add-to-cart, and revenue.

Both tools also offer similar reporting interfaces, with the standard functionality you would expect for analyzing an A/B test.

The VWO report is easier to understand, but the Convert report is more feature complete. Ultimately, both get the job done.

Winner: VWO 🎉

VWO is the winner, primarily due to the more powerful visual editor. Convert was a good runner-up, while GrowthBook is only currently suitable for a technical audience.

What Next?

I'm working on guides for setting up VWO for a Shopify store and running the most typical types of A/B tests using that setup. Follow/Subscribe here to get notified when those are ready!

I'd recommend you also check out our Experiment Club, if you want free help with setting up A/B tests every week.

If you'd love to run A/B tests but don't have the time, you could also check out our Experiment Service.

Happy testing!