The 13 Limitations of Google Optimize A/B Testing (ft. Hackle A/B Testing)

What is Google Optimize?

Google Optimize is a free online platform for users to use different tools to optimize their websites. Through optimization, Google Optimize helps users increase their visitor conversion rates and maximize customer experiences on their website. The main method used for optimization is experimentation and A/B testing, where Google Optimize helps users test out an alternative variation of an element on the website such as fonts, click-to-action buttons, banners, product images, and colors in order to see how they perform against a specific objective. A/B testing on Google Optimize essentially helps users find the best version of their website.

However, there are a few very important limitations with Google Optimize that prevent users from carrying out advanced website A/B testing. Moreover, as users start to conduct more A/B testing via Google Optimize, the more they will start to question the reliability of the data results within each A/B test.

Here at Hackle, we precisely understand the distinct needs that many users have with regard to A/B testing. We designed the right platform that not only accurately tracks data on unique users but provides users with the flexibility of designing the right A/B test required to adapt to each and every company’s unique needs.

The 13 Limitations of Google Optimize

1. Google Optimize affects the speed of the actual website.
2. Google Optimize takes time to load changes of an A/B test setting.
3. Google Optimize does not update A/B test results to the dashboard in real-time.
4. Google Optimize does not support SPA, Single Page Application websites.
5. Google Optimize offers a limited number of predetermined metric types.
6. Google Optimize doesn’t allow you to segment data results for each metric.
7. Google Optimize does not provide an advanced user targeting capability.
8. Google Optimize only offers website-based A/B testing.
9. Google Optimize counts the same user as multiple users.
10. Google Optimize samples user data in A/B tests.
11. Google Optimize has a limit to the number of metrics that can be set per A/B test.
12. Google Optimize has a limit to the number of A/B tests that can be run simultaneously.
13. Google Optimize does not support simultaneous A/B testing on both the production and development environments.

Google Optimize affects the speed of the actual website.

Google Optimize allows you to create your A/B test and directly create new variations by using the visual editor or a CSS code editor.

However, when A/B tests with different variations created with visual editors are accumulated in the system, the loading speed of pages will slow down, significantly affecting the user experience of those who visit your website.

However, using the A/B testing tools provided by Hackle and via Hackle’s SDKs to conduct A/B tests will never slow down the original speed of your service or homepage. To find out more about the importance of SDKs, click here.

You don’t want to compromise user experiences while conducting A/B testing.

Google Optimize takes time to load changes of an A/B test setting.

An SDK’s role is to periodically receive the set-up information or changes made from the dashboard and reflect the changes to your code.

Compared to Google Optimize, Hackle has a significantly shorter time cycle to update new configuration changes, allowing you to control all changes and ensure that all the updates to an A/B test occur in real-time.

Google Optimize does not update A/B test results to the dashboard in real-time.

As the data Google Optimize provides is obtained from Google Analytics, A/B test results, and data is collected from the previous day. In this case, it is difficult for the user to immediately judge and respond to the current situation.

Hackle updates the experimental results at least once an hour, so the time difference is managed within an hour.

Google Optimize is difficult to use for SPA or Single Page Application websites.

It is very difficult to conduct A/B testing with SPA (Single Page Application) websites with Google Optimize and often times you will realize that you are testing an unintended appearance of a variation for your website.

Exoscale provides the following definition of SPAs as “a website design approach where each new page’s content is served not from loading new HTML pages but generated dynamically through the ability to manipulate the DOM elements on the existing page itself.”

As SPA is a website that is not static and interacts with users by dynamically recreating the current web page, the appearance of objects on a SPA website can be unpredictable while using Google Optimize which primarily tests out two different static pages against each other. This could mean that A/B tests with SPA pages may not be implemented correctly and data may be inaccurately collected. Meanwhile, A/B testing with Hackle’s SDKs allows you to run seamless experiments for SPA pages safely and accurately.

Google Optimize offers a limited number of predetermined metric types.

There is a limit to the type of metrics you can set with Google Optimize. This means that you are limited in the ways in which Google Optimize calculates or formulates the results for each metric.

With Hackle, you are given the flexibility to select your desired event to place in the numerator and denominator of each metric. This allows you to put together and create a wide range of different types of calculation methods. For example, you can calculate the conversion rate as well as the average order amount (AOV), the average purchase price per user (ARPU), the average purchase price per purchaser (ARPPU), etc.

A filtering feature is also provided so that you can collect data for certain metrics on specific user segments.

Google Optimize doesn’t allow you to segment data results for each metric.

Hackle allows the segmentation analysis of metric results in an A/B test.

A segmentation analysis of each metric set in the A/B test can be conducted on the basis of the platform (iOS, Android, Web, etc.), browser, app version, or property information managed internally by your company (ex. membership member status, first purchase status, gender, age group, etc.)

The segmented metric results allow you to determine whether or not certain user segments are being affected more than others.

Google Optimize does not provide advanced user targeting capabilities.

Google Optimize does not provide more advanced targeting capabilities.

Hackle supports narrowly defined targeting conditions. You can set up several conditions and properties for each targeting category in order to conduct A/B tests on your very narrowly defined desired set of users.

Google Optimize only offers website-based A/B testing.

All platforms are supported by Hackle’s A/B Testing

Google Optimize only supports A/B testing on websites and not for mobile app environments. On the other hand, Hackle provides various SDKs to support the implementation of A/B testing on any mobile app, web, or server-based platform through SDKs provided in various languages.

Google Optimize counts the same user as multiple users.

Hackle’s User Identification VS Google Optimize’s User Identification

Google Optimize is based on Google Analytics where unique user/visitor IDs are not taken into account and the collected data is session-based. This is one of the biggest limitations of Google Optimize as the same visitor who visits a website twice with a span of 4 hours in between each visit will be logged as 2 different visitors on Google Optimize rather than 1 unique user. Google Optimize’s session-based data tracking makes it hard to obtain accurate results when calculating metric results.

Hackle’s A/B Testing takes into account “unique user counts” that you can easily manipulate.

Google Optimize samples user data in A/B tests.

Google Optimize samples data, which in some cases may provide statistically insignificant data.

Google Optimize results are dependent on Google Analytics. Unless you pay for Analytics 360, there is a high chance that your data is being sampled. Sampling is the practice of using a subset of the full traffic and reporting results derived from the sample set. Although sampling may lead to similar results as when the whole data set is being used, when you need precise data that you can trust such as the conversion rate of your website or the total revenue spent on a page, data sampling can be problematic.

The data for Hackle’s A/B Testing is not subjected to sampling.

Google Optimize has a limit to the number of metrics that can be set per A/B test.

Google Optimize allows for up to 3 pre-configured metrics per experiment. On the other hand, Hackle A/B Testing allows an unlimited number of metrics/objectives per experiment.

Google Optimize has a limit to the number of A/B tests that can be run simultaneously.

Google Optimize only allows a total of 5 different experiments to run at the same time, while Hackle allows users to set up an unlimited number of experiments to run simultaneously.

Google Optimize does not support simultaneous A/B testing on both the production and development environments.

Google Optimize does not separate the production and the development environment from the dashboard, and hence there is a possibility that developers will make mistakes. They may accidentally start an A/B test on the production environment that affects real users prior to testing out on the development environment in advance.

Hackle provides both the production environment and the development environment when conducting a single experiment, and hence this problem will not occur.

Check out Hackle at www.hackle.io in order to start creating your own customizable A/B tests and metrics that really matter.