Improving Planet 4 with A/B Testing
How, when and why P4 will implement data-driven decisions
The 2nd version of Greenpeace’s global engagement platform involves a focus switch from implementing and iterating the prototype to continuously improve existing features and integrate new functionalities. For both areas, the new mantra of the P4 team is “Testing before developing”
Among the multiple methods to test ideas and assumptions, the most effective one is A/B testing (also referred to as “split testing”), usually defined as “an experiment where two or more variants of the same page are shown to a distinct segment of users, while statistical analysis is used to determine which variation performs better and drive more conversions”.
Why A/B testing is the best way for P4
While being fairly easy to implement, this methodology plays a key role to improve P4 users’ experience and to achieve higher and deeper engagement, the ultimate goals of a platform whose concept is to drive people to action.
Besides the mere effectivity of the best alternative, other reasons to include A/B testing in our workflow are:
- Intuition is often wrong!
- Without hard data, it can be hard to justify changes
- Changes can result in unpredictable side-effects
- Most successful internet companies continuously AB test changes
- ~30% of the tests are successful. There’s a 70% chance that untested changes have no impact or negative impact
Source: Optimize Quick Tips Videos
A data-driven product improvement workflow
To make sure A/B testing becomes part of the continuous improvement daily practice and doesn’t just remain related to individual stories, the Planet 4 team decided to embed testing right into the feature request/improvement workflow:
The tool: Google Optimize
Part of Greenpeace’s global insights tool suite, Google Optimize covers most of our needs. Beside being natively integrated with Tag Manager, Analytics and Data Studio, it allows both international and local teams to test alternative versions of the Planet 4 design and/or content without touching child themes or customising CSS.
P4’s most recent data-driven improvement: a new cookies bar
The Planet 4 Roadmap includes some engagement features that depend on the acceptance of cookies, such as “view your impact” or “follow a campaign”. With this in mind, we started investigating how we could increase the percentage of users accepting the cookies in our websites.
1 — Preliminary analysis
To better understand which areas required major design improved, we investigated the conversion rates of the cookies banner in different P4 websites. Through this first analysis, we discovered that desktop conversion rates were pretty low, compared to mobile ones.
2 — New design proposal
Based on these findings, the design team worked with a larger and slightly darker cookies banner, especially revamped for desktop screens. The dev team then provided the css changes to apply on the A/B testing alternatives.
3 — Definition of methodology and goals
Since our objective was to increase the conversion rates, especially on desktop screens, we defined our primary KPI as the click-through rate on the “I accept the cookies” button. Key requisites for testing were that we would only target users without any cookies installed (to make sure that our conversion rate would be right) and that we would create two separate tests for each device (mobile/tablet vs desktop)
Before starting, we also defined our goal based on the first data analysis: aiming to increase by 10% the mobile conversion rate and by 40% the desktop conversion rate.
Bounce rate was set as a secondary goal for the test to make sure no side-effects or undesired loss of traffic would be caused by any of the options of the A/B test.
4 — Implementation
Our colleagues from Belgium and the Netherlands decided to take part in the tests, sharing the same Google Optimize container in both their production websites. Once tests were created and linked with the Google Analytics Global Property we were ready to start!
5 — Analyse the results
The test was live for 21 days and the results proved that the new design for the cookies banner should be adopted:
- Netherlands: 2–16% improvement on mobile conversion rate and 108% to 162% in desktop conversion rate
- Belgium: 6–33% improvement in mobile conversion rate and 163% to 268% on desktop conversion rate.
The new variant also had a better performance regarding the bounce rate, meaning that the test reached all our goals and was largely successful 🙌🙌🙌
Managing experiments across different websites
The work we are doing in testing the same design elements and features in different Planet 4 websites and its distinct audiences is only possible because of the Global Tracking Property. The adoption of the same standards for events and conversions is a must for those cross-region experiments and an opportunity for fostering a culture of knowledge sharing about testing and optimization.
Want to test with us?
We are already planning the next split tests for two potential P4 improvements, such as changes to the Take Action Floating Cards on blog posts and a new variation of the “Donate” button on mobile devices. If you are interested in joining any of those tests, don’t hesitate to send the team an email, comment below or tweet at #GPP4
