Improving Planet 4 with A/B Testing

How, when and why P4 will implement data-driven decisions

Julia Mente
Oct 7, 2019 · 5 min read

The 2nd version of Greenpeace’s global engagement platform involves a focus switch from implementing and iterating the prototype to continuously improve existing features and integrate new functionalities. For both areas, the new mantra of the P4 team is “Testing before developing

Among the multiple methods to test ideas and assumptions, the most effective one is A/B testing (also referred to as “split testing”), usually defined as “an experiment where two or more variants of the same page are shown to a distinct segment of users, while statistical analysis is used to determine which variation performs better and drive more conversions”.

A/B Test consists in a experiment where two or more variants of the same page are shown to a distinct segment of users
A/B Test consists in a experiment where two or more variants of the same page are shown to a distinct segment of users
Source: Braze.com

Why A/B testing is the best way for P4

While being fairly easy to implement, this methodology plays a key role to improve P4 users’ experience and to achieve higher and deeper engagement, the ultimate goals of a platform whose concept is to drive people to action.

Besides the mere effectivity of the best alternative, other reasons to include A/B testing in our workflow are:

  • Intuition is often wrong!
  • Without hard data, it can be hard to justify changes
  • Changes can result in unpredictable side-effects
  • Most successful internet companies continuously AB test changes
  • ~30% of the tests are successful. There’s a 70% chance that untested changes have no impact or negative impact

Source: Optimize Quick Tips Videos

A data-driven product improvement workflow

To make sure A/B testing becomes part of the continuous improvement daily practice and doesn’t just remain related to individual stories, the Planet 4 team decided to embed testing right into the feature request/improvement workflow:

Planet 4’s feature request workflow with A/B testing
Planet 4’s feature request workflow with A/B testing
Source: Planet 4 Team / Greenpeace

The tool: Google Optimize

Part of Greenpeace’s global insights tool suite, Google Optimize covers most of our needs. Beside being natively integrated with Tag Manager, Analytics and Data Studio, it allows both international and local teams to test alternative versions of the Planet 4 design and/or content without touching child themes or customising CSS.

Global Insights’ Tool Suite
Global Insights’ Tool Suite

P4’s most recent data-driven improvement: a new cookies bar

The Planet 4 Roadmap includes some engagement features that depend on the acceptance of cookies, such as “view your impact” or “follow a campaign”. With this in mind, we started investigating how we could increase the percentage of users accepting the cookies in our websites.

To better understand which areas required major design improved, we investigated the conversion rates of the cookies banner in different P4 websites. Through this first analysis, we discovered that desktop conversion rates were pretty low, compared to mobile ones.

Based on these findings, the design team worked with a larger and slightly darker cookies banner, especially revamped for desktop screens. The dev team then provided the css changes to apply on the A/B testing alternatives.

Image for post
Image for post
The resized cookies bar — desktop screen — Source: Planet 4 Team / Greenpeace

Since our objective was to increase the conversion rates, especially on desktop screens, we defined our primary KPI as the click-through rate on the “I accept the cookies” button. Key requisites for testing were that we would only target users without any cookies installed (to make sure that our conversion rate would be right) and that we would create two separate tests for each device (mobile/tablet vs desktop)

Before starting, we also defined our goal based on the first data analysis: aiming to increase by 10% the mobile conversion rate and by 40% the desktop conversion rate.

Bounce rate was set as a secondary goal for the test to make sure no side-effects or undesired loss of traffic would be caused by any of the options of the A/B test.

Our colleagues from Belgium and the Netherlands decided to take part in the tests, sharing the same Google Optimize container in both their production websites. Once tests were created and linked with the Google Analytics Global Property we were ready to start!

The test was live for 21 days and the results proved that the new design for the cookies banner should be adopted:

  • Netherlands: 2–16% improvement on mobile conversion rate and 108% to 162% in desktop conversion rate
  • Belgium: 6–33% improvement in mobile conversion rate and 163% to 268% on desktop conversion rate.
The results of Planet 4’s experiment with the Cookies Bar showed that the redesigned bar would drive a higher conversion rate
The results of Planet 4’s experiment with the Cookies Bar showed that the redesigned bar would drive a higher conversion rate
Source: Planet 4 Team / Greenpeace

The new variant also had a better performance regarding the bounce rate, meaning that the test reached all our goals and was largely successful 🙌🙌🙌

Managing experiments across different websites

The work we are doing in testing the same design elements and features in different Planet 4 websites and its distinct audiences is only possible because of the Global Tracking Property. The adoption of the same standards for events and conversions is a must for those cross-region experiments and an opportunity for fostering a culture of knowledge sharing about testing and optimization.

Want to test with us?

We are already planning the next split tests for two potential P4 improvements, such as changes to the Take Action Floating Cards on blog posts and a new variation of the “Donate” button on mobile devices. If you are interested in joining any of those tests, don’t hesitate to send the team an email, comment below or tweet at #GPP4

Planet 4

Planet 4 is the codename for the complete redesign of…

Julia Mente

Written by

Planet 4

Planet 4

Planet 4 is the codename for the complete redesign of Greenpeace.org. This project is developed using open principles, and this Medium Publication will be the central hub for news and updates pertaining to the Planet 4 project. Learn more at www.greenpeace.org/p4

Julia Mente

Written by

Planet 4

Planet 4

Planet 4 is the codename for the complete redesign of Greenpeace.org. This project is developed using open principles, and this Medium Publication will be the central hub for news and updates pertaining to the Planet 4 project. Learn more at www.greenpeace.org/p4

Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore

If you have a story to tell, knowledge to share, or a perspective to offer — welcome home. It’s easy and free to post your thinking on any topic. Write on Medium

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store