Shingles vaccine & dementia prevention: too good to be true?

A public peer-review

If you follow science news, you might have seen headlines claiming the shingles vaccine could reduce dementia risk. If true, this could be great news.

Dementia is a growing problem as the global population ages, and most people over the age of 40 are likely to need the shingles vaccine regardless of any benefits on shingles. What’s more, you can only get shingles if you have previously been infected with chickenpox, and in many countries a large proportion of the population under 40 has had the opportunity to be vaccinated against chickenpox. If the shingles vaccine prevents dementia by preventing later life harm caused by childhood virus infection, we could potentially see dramatic declines in dementia onset over the next 50 years.

So, it’s understandable why people were excited about these findings, reported in a pre-print at the end of May 2023. But importantly, pre-prints are essentially draft manuscripts. Researchers post these to get feedback on their methods and ideas, and to communicate quickly with other scientists. This study received attention on social media and in the news, but is not yet a finalized peer-reviewed publication. Peer-review is not perfect but it provides a chance for fellow scientists to give input on research, and (hopefully) identify errors or gaps that the study team missed.

Usually, peer-review is done privately, but given that the pre-print is available to anyone, some public peer-review might be useful. Below, I provide my own peer-review of this pre-print, structured the way I would write a peer-review if I were doing this for a journal. I have also shared (essentially) these same comments with the authors directly by email.

Why am I someone who can do peer-review for this paper? Well, I’m an epidemiologist who specializes in the methods for making sense of cause and effect, particularly from existing data like the current study. I also have training in infectious disease research, although I have not worked specifically on the shingles vaccine. Finally, I also have done some research on dementia.

What study is this? The paper is called “Causal evidence that herpes zoster vaccination prevents a proportion of dementia cases”, by Eyting et al. You can read it at: MedRvix May 2023

TL;DR: Below, I am sharing my full review. But some of you may just want to skip to the punchline. Is this study to good to be true? I argue, yes. Why? The method the authors used only works in a very specific set of circumstances, and those circumstances don’t seem to be happening here. For this analysis to be interpretable as telling us anything about the vaccine, there needs to be no difference in life circumstances between people born at the end of August 1933 and people born in the beginning of September that same year. But these groups *are* different and in an important way — children born Sept 2 1933 potentially received up to 12 months of extra education than children born August 31 1933. That’s a problem, because dementia and (less) education are almost certainly linked!

Overall comments:

This manuscript aims to assess the hypothesis that the herpesvirus varicella zoster may be a causal agent for dementia, via attempting to estimating the causal effect of shingles vaccination on (preventing) incident dementia among older adults in Wales.

The authors provide background to explain why they believe their analysis is estimating the effect of shingles vaccination but I have some reservations about whether these data allow assessment of this hypothesis & about the implementation of the method.

Background:

The authors make two related hypotheses.

Hypothesis 1: they hypothesize that the herpes zoster virus may be a cause of dementia in older adults. As we get better and better at collecting large datasets of biomedical data, we are learning more and more about how infections can cause long-term health problems. Even seemingly unrelated ones. The authors also provide as background details about laboratory studies in mice, as well as a current phase 2 trial assessing whether an anti-viral drug might help with treating mild Alzheimer dementia. So, given these studies & what we know of other viruses’ long term effects, the first hypothesis seems plausible. But there is a problem here — they don’t have data on exposure to or infection with the virus. So they can’t directly assess this hypothesis at all.

Hypothesis 2: The second hypotheses seems like a bit more of a jump to me, for a few reasons, but is the one they actually try to assess. This hypothesis says that receiving the shingles (herpes zoster) vaccine might prevent people from developing dementia. I don’t pretend to know enough about the biology here but this hypothesis doesn’t quite make sense to me. For one thing, only individuals who have already been infected with the herpes zoster virus need to get the vaccine — this vaccine does not prevent infection, but rather prevents reactivation of dormant virus that is already in people’s bodies. So, if this vaccine help prevents dementia it would be because *reactivation* of the virus later in life can cause dementia, or because chronic latent infection (with no symptoms) is a cause of dementia.

These two hypotheses also don’t necessarily line up the way we might think. For example, if the first hypothesis is correct, we might expect to find that individuals who report never having had chickenpox are also less likely to develop dementia. But these same people could also be less likely to get the herpes zoster vaccine and as a result, the overall group of unvaccinated people could be a mix of low dementia risk and high dementia risk individuals (again, assuming both hypotheses are true). This could result in the vaccine looking less helpful, or neutral or even harmful just because of who is getting the vaccine. If only the first hypothesis is true, then the unvaccinated individuals will include low-risk and average-risk individuals, whereas the vaccinated individuals would include mainly average-risk individuals (i.e., we might expect to reject the null for the second hypothesis if only the first hypothesis is true, not the second).

Mechanism of action: Another consideration is that the vaccine which participants in the current study received was a live-attenuated virus vaccine. This vaccine works to prevent reactivation of herpes zoster in much the *same way* exposure to children currently infectious with chickenpox provides zoster suppression. Unlike the US, the UK doesn’t currently require the chickenpox vaccine & so older adults are still quite likely to encounter chickenpox infections. If the live attenuated vaccine can prevent dementia by suppressing the virus, presumably contact with infectious children which *also* suppresses the virus would *similarly* prevent dementia. However, this seems somewhat in conflict with the observation that shingles (ie unsuppressed zoster) and dementia are both more common in women, despite women typically having more caregiving roles. Of course, this doesnt necessarily mean the hypotheses are wrong but it would be nice to see some discussion of these issues.

Potential Survivor bias: A second potential issue with studying this question is that dementia takes a long time to manifest, and people can be diagnosed at any age (although its more likely as people age). Many studies of dementia, including this one, start by looking at a group of people who have reached some threshold age alive and dementia-free. For this to be valid, the exposure must only cause the outcome after the threshold age, and the exposure must not affect the chances that people survive to the threshold age. Otherwise, the study population won’t be the same as the overall population. This is called survivor bias and is a potential issue here *if* the hypothesis is that the vaccine prevents dementia by counteracting damage from the virus.

If the virus causes dementia then we might expect at least some of the people who had been infected would have developed dementia prior to the start of this study — study participants are already quite old. On the other hand, if the hypothesis is that the vaccine prevents dementia independently of the any effects of the virus then survivor bias wouldn’t be an issue. The authors don’t discuss their hypothesis in enough detail to distinguish these.

Methods:

Study population: If we could conduct a trial to assess Hypothesis 2, it might involve recruiting people who are ~80 years old & who haven’t yet had the shingles vaccine (or shingles) & dont have dementia, and then assign some to get the vaccine and some to not & see what happens with dementia over the next several years. We wouldnt want to do this trial, but thinking about it makes it clear that the study population the authors chose is a good one for assessing their hypothesis— they look at adults ~80 years old without dementia or recent shingles & with no history of shingles vaccination. But, as mentioned above, potential survivor bias means this study population probably isn’t good for assessing the underlying hypothesis about the virus itself.

Exchangeability assumption (i.e., no confounding): Despite not being able to randomly assign the vaccine, the authors found a group where some individuals were eligible for the vaccine and others weren’t. They argue that these two groups are essentially random in all other ways. This is the key assumption upon which the results rely. The study population is adults age ~80 in Wales. For a group of adults born in 1933, they either became eligible for the shingles vaccine on or after Sept 2 2013 or they were never eligible. This was based on whether their 80th birthday happened Jan 1 to Sept 1 (never eligible) or Sept 2 to Dec 31 (eligible from birthday onwards). Although the authors evaluate a number of chatacteristics between the groups & find no major differences, I still have concerns about this assumption.

The authors argue that, within a small enough time window, a person’s birthdate is essentially random. And that may have been true in 1933 (although in modern days, we see trends in births across days of the week because of our ability to schedule giving birth!) The authors then also argue that a one week difference in birthdate is essentially equivalent to assigning people randomly to eligibility for the vaccine — that is, they argue that people born Aug 25 1933 to Sept 1 1933 have essentially indistinguishable lives to those born Sept 2 1933 to Sept 9 1933. For most pairs of weeks during the year, this would probably be a reasonable assumption. But not for these two weeks! The weeks on either side of Sept 1 are not equivalent weeks, and being born just before or just after Sept 1 can lead to vast differences in life trajectory. Why? In many places (including Wales in the 1930s), school entry eligibility depends on birth date, with children eligible to (& required to) begin school if they are at or above the threshold age on Sept 1st, but prohibited from entering school if they are below this threshold on Sept 1. This means that a child born on Aug 31 or Sept 1 could begin schooling *up to 12 months* before a child born on Sept 2 or 3rd. In contrast, children are generally eligible to leave school on or after their birthday when they have aged out of mandatory schooling (although, clearly most will stay through graduation). In the 1930s in Wales, the eligible age to start school was 5 & the minimum age to leave school was 14. So, children born Aug 25 to Sept 1 1933 were eligible to enter school in Sept 1938, and children born Sept 2 to Sept 9th werent eligible until at least Jan 1939 & in many cases not until Sept 1939. But all of these children could conceivably have left school by Sept 9th 1947. Thus, children born the week after Sept 2nd may have had up to 12 months less formal education than those born only one week prior. (See for example: https://onlinelibrary.wiley.com/doi/full/10.1111/sjpe.12247)

This is a potential problem because we know that years of education is a strong predictor of (decreased) dementia risk, and people who are never eligible for the shingles vaccine will likely have (at least slightly) more years education. The combination of those impacts could lead to making the effect of vaccination on dementia prevention seem smaller and so we might be inclined to believe the study results are a lower bound. But, if the underlying hypotheses about the virus are true, there’s another possible way for schooling to affect dementia — school is generally a major source of infection for children, so it’s feasible that children starting school earlier could be more likely to have ever had chickenpox. In this case, if hypothesis 1 is right, the older (and vaccine ineligible group) would have a systematically *higher* dementia risk. If this happens, the study could find a seeming effect of vaccination even if the entire effect is due to education-related infection risk not vaccination.

The above problem with the main assumption is the main issue with this study, but there are also some other more minor issues.

Sample size: The authors give the sample size for the entire study population (which is huge!), but the rationale for the method relies strongly on people born in a 2-week period in 1933 and they don’t tell us exactly how many people this is (it’s definitely a much smaller group!!). It would be very useful to have the numbers by birth week for the weeks during the main analytic window (this is in a supplemental graph but hard to read).

Statistical approach: The authors use a method called regression discontinuity. When the data meet the key assumption of randomness around the threshold, regression discontinuity can help us estimate the effect of an intervention within the narrow window of randomness. For this particular study, that means that if date of birth is essentially random in the couple weeks around Sept 1, 1934, the study analysis would tell us about whether the dhingles vaccine prevents dementia *for people born during those couple weeks in 1933*! Now, maybe that effect is the same as the effect of the vaccine for anyone born any time, but it also may not be! The 1930s and 1940s were a very turbulent time & there are potentially many differences in exposure to things like famine (war rationing), violence, displacement, interrupted schooling, etc, which could mean that what happened to people born in 1933 doesn’t match what happened to people born in later years.

Implementation of the analysis: The authors focus their assessment of the key exchangeability assumption on the pair of weeks around Sept 1 but, in fact, they don’t actually use either of these weeks in the main comparison! The data the authors have only includes *week of birth* (Mon to Sunday), and so the authors can’t actually identify whether someone was born Aug 25 to Sept 1 vs Sept 2 to Sept 9, 1933. (In 1933, the available weeks were Monday, Aug 21 to Monday Aug 27, Monday Aug 28 to Sun Sept 3, and Monday Sept 4 to Sunday Sept 10.) Since the authors cant assign births reliably to pre/post Sept 2 for the Aug 28 to Sept 3 week, they exclude people born in this week from their data. But this is a big problem. Because the people born in this week are precisely the people for whom they claim there is exchangeability. Outside that week, even the authors’ arguments for exchangeability get weaker & weaker. It might be reasonable to argue that (if exchangeability exists), the assumption still holds for some small period of time on either side of this week. The analysis implicitly makes this assumption but the authors do not discuss it in depth. They also do not choose the window directly, either, relying on model fit and as a result ending up with an interval of 90 weeks. It is somewhat unclear whether this is 45 weeks on either side of the excluded (main effect) week (90 total weeks) or if it is 180 total weeks (90 on either side). Either way, this means the analysis requires that we now assume that being born before Aug 28 or after Sept 4 1933 is random for babies born during a multi-year period — a much less reasonable assumption!

Bottom line: At best, this study might suggest dementia onset risk can be lowered by either the (live-attenuated) shingles vaccine, or education, or infection with chickenpox, or other factors related to duration of schooling, or any combination of those. My money’s on education & schooling.

Final thoughts: While I don’t find this study convincing, that doesn’t mean the hypotheses the authors test are wrong — just that this study doesn’t help us decide whether or not it is right. It’s too soon to really ask this question & beyond the scope of this paper but if the hypotheses here are true, we might expect dementia rates to increase (with an appropriate lag) in the US following chickenpox vaccine was introduced (because of reduced exposure to infectious kids), with modulation by rates of shingles vaccination, and then ultimately decline as those vaccinated against chickenpox age that the rate of dementia may decrease. A similar patter would not be expected in the UK where chickenpox vaccination is less common. It will be interesting to observe these long-term trends to see if this plays out.