Are 7 French fries too many?
A causal inference explainer
We all love French fries, but recently a Harvard professor made headlines with the claim that we should only eat 6 fries per serving.
Less is more is generally good advice when it comes to any food, but why is 6 fries the magic number? Why not 5 or 7? Unfortunately, there is no probably no scientific way to know the exact number of French fries we should eat in one serving!
Even worse, all the reasons why we can’t know if 5, or 6, or 7 fries is better are also reasons why it’s very hard to know whether French fries are even bad for us at all!
And worst of all, these reasons also apply to almost anything else you want to study! If you are interested in making decisions based on evidence, then grab a 🥗 & get comfy! This explainer on estimating causal effects using the target trial framework is for you!!
To begin, let’s focus on a specific evidence-based decision. Imagine you want to reduce your intake of French fries with the specific goal of reducing your lifetime chance of having a heart attack.
This decision is really two separate decisions: how often should I eat any fries; and how many fries should I eat in a serving?
To help you live your best life (i.e. allow you to eat the maximum safe number of fries), we need to ask a pair of questions:
- What is the best frequency of French fry consumption to prevent heart attacks?
- What is the best serving size of French fries to prevent heart attacks?
If we had unlimited resources, the best way to answer both these questions would be to conduct a randomized trial. Even though we don’t really have unlimited resources, we can use the trial we would like to do to help design our analysis. We call this the target trial approach.
Let’s start by designing a target trial. Randomized trials have five main components:
- Who should we include? (eligibility)
- What treatment or exposure strategies should we compare?
- How long should we follow the people in our study for?
- What outcome should we compare and what question about the outcome do we really want to answer?
- How should we design our analysis to answer this question?
First, we need to answer to decide whether we want to know the best fry eating strategy for you to follow for your whole life, or the best strategy for people your age to start following?
Since we’re trying to help you decide if you should reduce your fry eating, let’s assume we want to know about the optimal fry eating strategy to start following right now. Since I don’t know who you are, let’s say you just turned 18.
Should we exclude anyone from our trial? If we want to know about preventing a 1st heart attack, we should exclude those who have already had one, but if we want to know about any heart attack then everyone can join.
But what about people who have never eaten fries? That depends on our causal question…
Do we want to know the best fry eating strategy for to adopt on your 18th birthday, or do we want to know the best strategy for only people who’ve been regularly eating fries to adopt on their 18th birthday?
The first question is probably much more relevant to you! So, the first step of our target trial (eligibility criteria) is sorted out: we’ll enroll a bunch of 18 year olds who have never had a heart attack.
Next, we need to decide on the strategies we’d like to compare. What are our target interventions?
We want to know about the frequency and amount of French fries, but what do we mean by French fries? Does it matter if we eat shoestring fries, curly fries, home fries, tater tots….? It might! This kind of detail is important to specify, so let’s assume we meant shoestring fries.
To determine the maximum safe fry-eating strategy, we want to know about a joint intervention on the frequency & amount of fries. That is, we want interventions that look like this: eat shoestring fries Z times per month & eat exactly X shoestring fries at once sitting.
In our hypothetical target trial, where we have unlimited resources, we can specify ranges for Z and X. Say, Z = 0 to 30 and X = 0 to 100.
Then we can randomly assign some people each combination of Z and X and see which group has the lowest chance of a heart attack.
But 100*30 = 3000 trial arms. That’s an awful lot of people! 😬
For now, let’s not worry about the sample size, and move on to the next question: when should the trial start and how long should the trial last?
We already decided to start when participants turn 18 — in our hypothetical trial, we could randomize them on their birthday. But how long should we follow them for?
Well, the average age of 1st heart attack for men in the United States is ~66 years old and for women is ~70 years old. So we probably want to follow our trial participants until then. To make the numbers easy, let’s follow everyone until their 78th birthday— that’s 60 years of follow-up!
Good thing we have those unlimited resources!
The fourth component of our randomized trial is the outcome and (causal) question of interest.
We already decided our outcome will be heart attacks, but we could mean first heart attack, total number of heart attacks, fatal heart attacks, non-fatal heart attacks… all of these details need to be decided. To make things easy for ourselves, let’s say we want to know about first heart attack.
We still need a causal question — in particular, we need an effect or estimand that we want to try to estimate.
In randomized trials, we usually like to estimate the intention-to-treat effect: the heart attack risk if everyone was assigned to a specific amount & frequency of fries versus if everyone was assigned to no fries.
There’s a problem thought. The intention-to-treat effect can under- or over-estimate the true effect if some participants drop out or don’t follow their assigned fry-eating strategy exactly.
But, we’re asking people to eat X amount of fries Z times per month for 30 years! No way everyone is going to follow that exactly, and it’s almost certain that in the real world some people would drop out of our study. So, the intention-to-eat effect isn’t going to be very useful.
The other option is a per-protocol effect: heart attack risk if everyone ate the amount & frequency of fries we told them to and stayed in our study.
Per-protocol effects are more complicated to estimate, in large part because we need people to record their actual fry eating behavior for 60 years! But (in theory) it can be done!
Two important caveats on how to estimate the per-protocol effect in our trial. First, we absolutely require information on the predictors or fry eating & heart attack over time — called confounding. And second, we must use a special type of statistical analysis called g-methods if past fry-eating ever predicts future fry-eating, which it almost certainly does! (Follow me for articles about how g-methods work!)
Okay, now we have a basic target for finding the fry-eating strategies for 18-year olds to adopt that minimizes 1st heart attack risk.
Who wants to fund my 3000-arm trial w/ 60 years of follow-up & daily fry consumption diaries, plus regular measurement of confounders?
No takers?? 😬😬😬
Okay, don’t panic, we may still be able to answer our question: let’s do an observational study! We can use our target trial to design it, and we probably don’t even need to change much!
Let’s recap the trial: enroll 18-yr olds; randomize; record fry eating, health, behavior, & heart attacks for up to 60 years; adjust for post-baseline confounding with g-methods; & learn best fry eating plan based on per-protocol effect.
What’s different in our observational study? Just that we don’t randomize! This means three things.
First, we have to worry about confounding before randomization (called baseline). But that’s hardly a big deal — we were already dealing with 60 years of confounders to estimate our per-protocol effect, so what’s one more time point?
Second, even if we wanted to know the intention-to-treat effect in the target trial, the observational data can only be used to estimate the per-protocol effect. That’s because the intention-to-treat effect is the effect of randomization, but we haven’t randomized anyone.
The 3rd issue could be tricker. We wanted 3000 trial arms to cover all the possible combos of fry eating frequency and amount, and that was too many for our trial to be reasonable. But our observational data will also only work if there are people who eat every fry frequency and amount combination!
That is, we need data on all 3000 strategies to estimate the heart attack risk under all of those strategies. Plus we need something called positivity: we need everyone in our study to have had a chance of following every fry-eating strategy!
Let’s think back to how this all started — we wanted to know if 6 fries was the right amount. To decide between 5 or 6 or 7 fries, our biggest problem is probably going to be that there almost certainly aren’t many 18+ year olds out there who only ever eat exactly 5, 6, or 7 fries in one sitting!
And even if we’re willing to change our question to something like ‘eat less than 50 fries in each serving’ to ‘eat 50 to 100 fries in each serving’ , how many of you count your fries ever, let alone every time?!
We started off so well! A target trial that could tell us exactly what we wanted to know! But it required lots and lots and lots of data and extremely high participant engagement, and we just can’t afford that.
We made one teeny change that we thought could help (giving up on randomization in exchange for controlling for baseline confounding) but our observational study still fell apart because we want to know about things no one does, and even when people do them, we can’t measure well!
Take home message? This example shows us two key benefits of the target trial approach.
First, for realistic strategies that we can measure, a target trial helps us design an observational study that will work.
And second, when our strategies aren’t realistic, the target trial framework can help us figure that out before we’ve spent all the time and effort and money doing a study that won’t help us make a good decision!
Thanks for reading! This article started life as a twitter thread filled with gifs, which you can read here: french fry #tweetorial.
If you’d like to know more about target trials, you can read more about them in this open access paper: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4832051/
If you want to know more about causal inference, follow me on here and on Twitter Ellie Murray. I tweet and blog about methods for causal inference that can help you make better data-informed decisions.
