Using Causal Inference for Measuring Marketing Impact: How BBC Studios Utilises Geo Holdouts and CausalPy

Published in

BBC Studios Data and Engineering

7 min readMar 18, 2025

Introduction

Marketing teams constantly grapple with the challenge of accurately measuring the effectiveness of their campaigns. Traditional A/B testing is widely regarded as the gold standard for causal inference, allowing marketers to compare exposed and unexposed audiences in a controlled manner. However, A/B testing is not always feasible — particularly for out-of-home (OOH) advertising, multi-channel marketing and broad-reach digital campaigns where audience exposure cannot be strictly controlled.

To address this challenge, causal inference techniques provide robust methodologies for estimating marketing impact when controlled experiments are impractical. Among these, Bayesian Synthetic Control methods, implemented using CausalPy, offer a statistically sound approach for conducting quasi-experiments. At BBC Studios, we leverage these methods to assess the effectiveness of our OOH marketing efforts, providing data-driven insights when randomised control trials (RCTs) are not an option.

The Limitations of RCTs in Multi-channel Marketing

RCTs are widely used in digital marketing, where users can be randomly assigned to treatment and control groups based on their cookies or device IDs. However, in the context of OOH advertising and when we using multiple marketing channels in your efforts (Google, Meta, YouTube etc), implementing RCTs poses several challenges:

Lack of Exposure Isolation: Unlike digital campaigns, where a control group can be explicitly excluded from seeing an ad, OOH campaigns expose consumers in the same geographical region to the treatment, making isolation difficult.
Cross-Platform Contamination: Consumers interact with multiple marketing channels simultaneously, such as online ads, television, and social media. This makes it difficult to ensure a user is not exposed to the intervention when they are meant to be into be in a control group
Logistical Complexity: Unlike digital advertising, where ads can be toggled on or off for specific audiences, OOH campaigns involve physical placements (billboards, transit ads), making selective exposure unfeasible.

Given these challenges, quasi-experimental approaches — such as Geo Holdouts combined with Bayesian Synthetic Controls — offer an alternative solution for estimating causal effects in the absence of an ideal RCT setup.

Bayesian Synthetic Controls for Geo Holdouts

To overcome the limitations of traditional A/B testing in multi-channel marketing, we implement Bayesian Synthetic Control methods using CausalPy. This approach allows us to construct a credible counterfactual — i.e., what would have happened in the absence of the marketing intervention — by leveraging data from untreated geographical regions (Geo Holdouts).

How Bayesian Synthetic Controls Improve Traditional Synthetic Control Methods (SCM)

Traditional Synthetic Control Methods (SCM) estimate treatment effects by constructing a synthetic control group from untreated units (e.g., regions where the campaign was not run). However, these methods rely on deterministic, frequentist-based approaches that can be sensitive to data limitations and assumption violations.

In contrast, Bayesian SCM introduces probability distributions to model uncertainty in:

Synthetic control weights (how much each control region contributes to the estimated counterfactual).
Outcome predictions (allowing for confidence intervals rather than a single deterministic point estimate).

This approach improves the robustness and interpretability of causal impact estimates, especially when working with small sample sizes or highly variable market conditions.

How Geo Holdout-Based Bayesian Synthetic Control Works

1. Creating Geo Holdouts

To conduct a valid quasi-experiment, we select specific geographical regions where the campaign will not be run. These regions serve as the control group.

Control regions are chosen based on their historical similarity to the treated regions (e.g., population demographics, past engagement, and sales patterns). We elected to chose Geolocations that may be affected by the same localised news patterns and seasonality, whilst not being so local that people may be exposed to the treatment.
These untreated regions collectively form a synthetic control group, which is weighted based on their relative contribution to the overall market dynamics.

2. Constructing the Synthetic Control

Using historical data, we identify a weighted combination of untreated regions that best mimics the pre-campaign trends of the treated region.

This step ensures that, prior to the intervention, the synthetic control behaves similarly to the treated market.
The Bayesian framework adjusts for noise and external market conditions, making the counterfactual estimate more robust than deterministic approaches.

3. Estimating the Incremental Impact

Once the synthetic control is established, we measure the difference in post-intervention performance between:

The treated region (where the campaign was run).
The synthetic control (which represents the counterfactual scenario without the campaign).

By comparing these two, we estimate the true causal impact of the OOH campaign, ensuring that the results are not confounded by seasonality or macroeconomic shifts.

Implementation at BBC Studios Using CausalPy

At BBC Studios, we utilise CausalPy, an open-source Python library, to perform Bayesian causal inference for OOH marketing campaigns. The implementation involves:

1. Pre-Treatment Data Collection

Gathering historical traffic, engagement trends, and sales data from both treated and control regions.
Identifying macroeconomic factors that could influence results, such as seasonality, competitor activity, or major events.

2. Control Region Selection

Selecting geo holdouts that exhibit high correlation with the treated region before the campaign begins.
Ensuring that demographics, purchase behaviour, and media consumption patterns are aligned between the treated and control regions.

3. Bayesian Weighting Assignments

Using Bayesian inference to determine how much each control region contributes to the synthetic control.
Unlike traditional SCM, Bayesian methods allow for adaptive weighting based on uncertainty and external influences.

4. Model Validation

Assessing the model’s ability to predict pre-campaign outcomes, ensuring that the synthetic control is a valid counterfactual.
Adjusting for seasonal trends, competitor interventions, and macroeconomic factors.

5. Causal Impact Estimation

Once the campaign is launched, comparing post-treatment results between the treated region and the Bayesian synthetic control.
Generating credible intervals around the causal estimates to quantify uncertainty.

Below is an example code snippet, where we implement geo-holdouts based on country as the independent variable and revenue-per-user (RPU) as the dependent variable:

import numpy as np
import pandas as pd
import causalpy as cp
import matplotlib.pyplot as plt

# Set random seed for reproducibility
np.random.seed(42)

# Define time range
date_range = pd.date_range(start="2023-01-01", periods=100, freq="D")

# Define intervention point
intervention_date = pd.Timestamp("2023-04-10")

# Generate synthetic revenue per user (RPU) data for control countries (before intervention)
def generate_control_rpu(base_value, trend=0.01, noise=0.1):
    return base_value + trend * np.arange(len(date_range)) + np.random.normal(0, noise, len(date_range))

# Control countries data (pre- and post-intervention)
control_countries = ["Germany", "Spain", "Italy", "UK", "Netherlands"]
control_data = {country: generate_control_rpu(1.5 + i * 0.05) for i, country in enumerate(control_countries)}

# Generate treated country (USA) with an intervention effect
usa_rpu = generate_control_rpu(1.6)
intervention_index = np.where(date_range == intervention_date)[0][0]

# Apply intervention effect to USA (increase in revenue per user post-intervention)
usa_rpu[intervention_index:] += 0.3  # Assume a +0.3 uplift post-intervention

# Create DataFrame
df = pd.DataFrame(control_data)
df["USA"] = usa_rpu
df["date"] = date_range

# Convert data to long format for CausalPy
df_long = df.melt(id_vars=["date"], var_name="country", value_name="rpu")

# Convert data to wide format (needed for CausalPy)
df_wide = df.pivot(index="date", columns="country", values="rpu")

# Define pre- and post-intervention periods
pre_period = (df_wide.index.min(), intervention_date - pd.Timedelta(days=1))
post_period = (intervention_date, df_wide.index.max())

# Run Bayesian Causal Impact using CausalPy
model = cp.pymc_experiments.BayesianCausalImpact(
    df_wide, 
    treatment_column="USA",  # The treated country
    pre_period=pre_period,
    post_period=post_period
)

# Fit the model
model.fit()

# Plot the results
model.plot()
plt.show()

Challenges & Considerations

While Bayesian Synthetic Control methods provide a powerful framework for estimating causal effects in marketing, they come with practical challenges:

Data Requirements: Robust historical data is essential to create an accurate counterfactual.
Computational Complexity: Bayesian methods require significant computational power, especially when dealing with high-dimensional data.
Attribution Complexity: Isolating the impact of OOH from other marketing channels (e.g., TV, digital, social media) remains an ongoing challenge.
Coordination from marketing teams: Syncing with teams internally to holdout areas adds an overhead in terms of business planning and coordination.
Sensitivity to Seasonality: Careful modelling is required to adjust for seasonal variations that could distort causal estimates.

Conclusion

When traditional A/B testing is impractical for measuring marketing impact — particularly in OOH advertising — Bayesian Synthetic Control methods provide a rigorous alternative. By leveraging Geo Holdouts and CausalPy, we can construct reliable counterfactuals, estimate incremental campaign lift, and quantify uncertainty in causal estimates.

At BBC Studios, these methods enable us to make data-driven marketing decisions, ensuring that our investments in OOH advertising are both measurable and scientifically validated.

BBC Studios Limited is the commercial subsidiary of the BBC, established in April 2018 by merging its production and global distribution divisions. As a British content company, we operate independently from the BBC’s public service, delivering original content and experiences to audiences worldwide.

Interested in joining us? Register on the careers website and search “BBC Studios” or data — we recruit both UK and worldwide. We are on the lookout for talent from apprentices to senior roles.