Is science really self correcting?
Something old and older
We teach a small group learning course on critical appraisal for MPH students using a unique format. Each group is divided into three smaller subgroups of four or five students. The first subgroup receives only an introduction to the problem, and without looking at the chosen article, discusses their choice of study design to address the problem. The second subgroup receives only the published abstract and comments on what details they would expect to see in the full article. The third subgroup reads the whole article and in addition to highlighting its strengths and limitations tries to answer queries raised by the first 2 groups. The goal is not to simply trash an article (as seems to be the goal for many journal clubs) but to improve critical appraisal skills while thinking about research questions, designs, and the necessary compromises that are often required in research.
One of the instructors (for reasons that will become evident later) chose an older NEJM article on the effectiveness of influenza vaccine in the elderly to evaluate. Despite several strengths including a prospective cohort design with a large sample size and controlling for several confounders, the reported 50% reduction in all cause mortality for the vaccinated group was a red flag for the presence of a substantial bias.
Letters to the editor immediately following the article’s publication noted that the mortality benefit exceeded all reasonable estimates, as influenza mortality during the winter months typically accounts for 5% or less of total mortality. The perceived bias was attributed to residual and unmeasured confounding. While there may well be some degree of residual confounding, it seems unlikely that it could explain this magnitude of bias, from a theoretical ceiling of an approximate 5% mortality decrease to the reported 50% decrease. The clue to resolving this bias is found in the following sentence;
“All noninstitutionalized members of the plans were included in that season’s cohort if they were 65 years of age or older as of October 1, had been continuously enrolled in the plan for the preceding 12 months, were alive on the first day of the influenza season, and were either continuously enrolled or died during the outcome period.”
Since it is virtually impossible that the vaccinated individuals all received their vaccination on Oct 1 of each flu season, the researchers must have looked into the future to determine vaccination status, leading to exposure time misclassification. The bias associated with neglect of the period of exposure to risk was described by Hill in 1937 Hill, Lancet 1937 Principles of Medical Statistics. Specifically in Table XVII, reproduced below, Hill provides a numerical example demonstrating that ignoring the time of vaccination can result in a fallacious relative risk reduction of 50%, even when the data has been simulated with equal mortality rates.
Inspired by Hill’s example, I re-examined the 2007 NEJM data. Lacking details on the timing of vaccination and the associated deaths rates according to exposure group forces certain assumptions. On Oct 1 nobody was vaccinated but it seems safe to assume that the vaccinated didn’t all receive it immediately. Rather I assumed on Nov 1, Dec 1 and Jan 1, 1/3 got vaccinated at each date with 1% waiting until Feb 1. Following Hill’s example, I assumed a constant death rate per month of 0.002058 that is independent of vaccine status producing this Table which is consistent with the published data.
For the first month all person-time belongs to the unvaccinated group and so do all the deaths (713872 * .002058 = 1469). For the next month, 1 /3 of the final 415,249 vaccinated individuals (.33 * 415249 = 137032) had now been exposed and their deaths (137032 * .002058 = 282) are attributed to the vaccinated group. These vaccinated individuals and those that died in the first month are then removed from the unvaccinated pool and the constant mortality rate (.002058) is then applied to the remaining unvaccinated individuals (575371). The same procedure is repeated for each subsequent month.
As the Table demonstrates despite a constant mortality rate across exposure groups if one analyses the data, as per the original NEJM analysis (see unadjusted RR in the Table), the crude RR = 0.45. This is slightly different than the NEJM reported RR for mortality but is to be expected as I had to assume exposure timing and individual group mortality rates. I’m not claiming this is what transpired only that without the actual data this could have happened. The essential point is that an unadjusted RR = 0.45 can arise from a data simulation that assumed a monthly mortality rate that is NOT different between the exposed and unexposed groups. In other words, almost the entire NEJM mortality effect could be explained by inaccurate classification of exposure times. Using an appropriate method of person time, gives the correct simulated RR = 1, exactly as Hill demonstrated 87 years ago!
In more modern times, this inaccurate accounting of exposure time and analysis has been termed immortal time bias by Suissa, one of the co-instructors for our critical appraisal course and who suggested the article for review. Suissa has shown that immortal time bias can lead to large discrepancies from the truth by either misclassification or selection bias, often dwarfing bias due to unmeasured or residual confounding.
Hernan and colleagues have proposed a paradigm of emulating a randomized trial to minimize the biases associated with observational studies, including immortal time. The key is align i) Elgibity ii) Assignment and iii) follow-up times (t=0). Failure to align, as shown in the Figure may lead to i) prevalence bias ii) selection bias iii) immortal time (due to selection bias) and iv) immortal time bias (due to misclassification).
If eligibility and assignment happen before t=0 then classic prevalence and selection biases may occur. If assignment or eligibility are determined in the future after t=0 then the biases are referred to as immortal time bias (ITB).
The 2007 letters to the editor queried the unbelievable mortality result but nobody brought up this potential source of error, rather uniquely blaming residual confounding. It’s not only the original authors, reviewers and editors but the whole medical commons who seem oblivious to this bias. Moreover, as of today, despite these initial confounding concerns, this publication has been cited 488 times, including 22 times in 2023 and 2024 with no mention of this potential bias. As an example, in 2023 we can find the following regarding vaccinations “prevent influenza-related hospitalizations and deaths among older adults” citing this 2007 publication as the reference.
So what are the take home messages?
First, teaching provides a great way to review and to synthesize our understanding. Second, answers to current problems can often be found by reviewing past foundational work. Third, when searching for biases, don’t stop with confounding and in the presence of a large unexpected effect size, think immortal time bias. Finally, while science may be self-correcting, this path can be incredibly long and winding (an academic version of this article was refused by more than a half dozen medical journals, including medRxiv).
