26 March 2021 | Advertising, AI and Machine Learning, Audience, Causality, Consumer journey, Customers, Data, Data Science, First-party data

For those responsible for online content of any kind, editorial or advertising, one of the great tools to use — to find out what works best by determining cause and effect — is A/B testing. Unsurprisingly, use of A/B testing is growing fast as publishers and advertisers seek to raise levels of user engagement with their pages and ads.

A/B tests are controlled experiments involving an existing content layout (editorial/advertising) that is shown to a Control group of users, and a variant layout that is shown to a Test group. Users are randomly split (persistently) between the two groups. The process requires decisions on what to test (the hypothesis), the user sample size needed for a robust result (within a confidence level), the test period required to achieve the sample, and the outcome metric for the test.

A/B testing can be used to determine the effect of a wide range of content changes, such as:

  • Editorial: copy, length, headlines, fonts, colours, layout, image, video, links etc.
  • Advertising: ad copy, size, format, colour, font, image, video etc.
  • Ecommerce: page layout, navigation, copy, offers, checkout, images, call-to-action etc.

An issue with A/B testing is that it can take a long time to discover content changes that make a predictive and significant difference (signal rather than noise). Many tests are cull-de-sacs, with lower or no discernible effects. When tests are run sequentially the discovery process can be slow and laborious.

A potential solution is to deploy machine learning that automates iterative and concurrent testing of content changes. For example, a font/colour change or copy/format change can be tested using a matrix of possible combinations. The automated system could iterate test combinations, and concurrently evaluate multiple tests through automated selection of user samples. This much faster approach can discover content change combinations that have significant predictive effects.

From an advertiser perspective, the best outcome metric for an A/B test is sales. Advertisers want to know the effect of editorial and advertising changes on their sales. However, this requires combining editorial + advertising + ecommerce into a single ‘closed loop’ system to determine the effect of context/ad changes on sales. To close the loop, the system needs visibility of users across editorial, advertising, and ecommerce environments.

As third-party cookies come to an end, look out for the companies best placed to use their own first-party data (customers with log in identities) to maximise the effect of combined editorial, advertising and ecommerce environments. Machine learning may soon discover new rules for how brands grow.




Scientific Advertising is a compilation of short posts on the science of how brands grow

Recommended from Medium

Converting a PySpark DataFrame Column to a Python List

Optimvest BTC report 28/05/21

Take Scenario Planning to the Next Level

Using SVM With Kernels To Predict Whether A Client Will Subscribe A Term Deposit

Working with data: New York Times ‘new words’ dataset

Linear Regression From Scratch

How to flaunt your Passion for analytics in Data science job interviews?

Machine Learning, the way I’m learning(II)

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Frank Harrison

Frank Harrison

I am a researcher, data scientist, consultant, and owner of Croft Analytics — see https://www.croftanalytics.com

More from Medium

Machine learning is the future of marketing

machine learning is the future of marketing

Bad data, its effects on marketing, and navigational strategies

marketing intelligence, data analytics, marketing analytics, online customer data

Martech Interview with Brian Gilman on Customer Experience

University-Business Collaboration