Nootropics for Dummies #2: EVIDENCE—Having It

9 min readMay 14, 2019

In this multi-part, opinionated, skeptic’s guide to the world of nootropics and supplements at large, I attempt to offer some context and perspective on a broad, controversial topic. This is part 2: EVIDENCE-Having It. If you didn’t read part 1, here’s the link.
Necessarily, in the interest of expedience, some of this will be a gloss. Hence the ‘dummies’ bit in the title. I don’t think you’re a dummy — it’s a figure of speech. Please do read on, regardless of your self-assessed intelligence.

EVIDENCE - Having It

To this point, we’ve just barely covered what a nootropic is, but it’s time to address the elephant in the room: the supplement industry has a trust problem. If you’re anything like most people, like me, you have mixed feelings about supplements.

Even after years of research and inside perspective on the industry, I still see supplements as a mixed bag. While the FDA does regulate many things about supplements, strictly speaking there is little to prevent someone from selling just about anything as long as it’s not mislabeled or marketed with unsubstantiated claims. Beyond that, due to limited enforcement resources, bad actors can do otherwise until they’re caught. We’ll talk more about regulation in a later post in this series.

With that said, there are supplements out there that can help with all sorts of things. Supplements are, in the eyes of the FDA, food, and in the same way that changing your diet can have an effect on your health, your focus, your athletic performance, your sleep, or your skin, so, too, can some supplements.

So, simply at the level of ingredients alone, how can we tell what works?

You may not like this

The answer, depending on how research-oriented you are, might not please you. You guessed it — it’s research. If you believe in science (and I hope you do) and you believe in modern medicine (I also hope this for you), then you have some tremendous resources available to you for evaluating the efficacy of a given ingredient.

RCTs: The Gold Standard

At a minimum, what we want to see are Randomized Controlled Trials, or RCTs, considered to be the ‘gold standard’ for clinical research. To simplify wildly, a group of subjects are randomly divided into two or more groups — a group that receive some treatment and groups of those that either don’t or perhaps receive a previously-studied treatment (the ‘control’ or placebo group(s)).

When reviewing this article, a physician informed me that it is, strictly speaking, wrong to say that the control groups who receive placebo receive no treatment because ‘that effect is real and the research about placebo is so cool’. I have to agree, and I intend to learn and write more about placebo in the future.

Ideally, nobody, including those conducting the experiment, know who is in which group. These are referred to as ‘double-blind’ studies, although that term is a bit ambiguous (more on blinding below.)

A note: one of my science advisors helpfully pointed out that someone has to assign participants to groups so the findings can be connected back in the end, and this person is necessarily not ‘blinded’. Lest that seem impossible to you — take note.

As you might guess, it’s a little more complicated than that, though, and a study does not have merit simply by dint of being an RCT — an RCT design is simply the best-known structure for a clinical study — the table stakes, if you will.

Caveats, Bias, Misreadings, Blinding, and Gotchas (Validity)

This is where we get a bit into the weeds. For the TLDR on this — know that not all trials are created equal. If you want to know more, you have to dig a bit, or at least find a source you trust to summarize and comment. Feel free to skip ahead to the next section for some suggestions of where to find higher-level analysis.

In evaluating the usefulness and accuracy of the conclusions presented in a study, we need to look out for a number of confounding factors, inaccuracies, opportunities for bias, statistical error, and conflicts of interest.

Trial design is a multi-course topic for medical students and a rich field of study in itself, so we’ll have to continue to gloss, but here’s a short list among many things that might compromise the quality of a study, or at least suggest that a grain of salt accompany their conclusions:

Structural Compromise. For cost and design reasons, trials are sometimes conducted with a change that compromises the basic RCT structure. As an eg, controls might be inadequate to provide a baseline, or absent entirely, leaving it unclear how the measured effect relates to placebo or confounding conditions. Studies are expensive, but it’s not only to save money that researchers may fail to structure a study perfectly — sometimes it’s borderline impossible to execute a true RCT (examples below) — even if the reasoning for the compromise is sound, the fact remains that compromises are likely to introduce uncertainty.
Selection and Randomization Bias. The selection of participants, and the randomization thereof, is a place where bias (a short list) can be introduced to the study, albeit unintentionally. As an example, let’s say the researchers assume age to be a factor in the outcome of the trial, and thus distribute participants otherwise randomly except by age, which they intentionally distribute evenly across the test groups. This is known as stratified randomization. In so doing, they now need to control for other factors, like a difference in the rate of post-surgical healing, for example, that are related to age, which, as you can imagine, is a a tough thing to do without introducing even more uncertainty. Randomization is a whole statistical ball of yarn unto itself.
Blinding Bias. Some studies can be difficult to blind properly —for proper blinding, participants, clinicians, data collectors, outcome adjudicators, and data analysts should all be properly blinded. Say, for example, that a study is looking at outcomes of surgical intervention, and a part of the analysis requires physical examination of patients or looking at x-rays … how do you keep physicians or radiologists from immediately knowing who among the patients had surgery and who didn’t? If you get creative and, say, redact parts of an x-ray, how can you be certain that hasn’t introduced bias itself? You’d need to do another study on the blinding method! Proper blinding itself is hugely important to the outcome of a study: one meta-analysis of 250 RCTs found that trials that reported double-blinding found a 17% difference in the size of the estimated treatment effect versus those that did not double-blind. This has been confirmed in other studies.
Attrition Bias. Significant numbers of participants can drop out of one or both arms of a study, and sometimes the reason participants left the study is significant but unaccounted for, or the distribution in the remaining group could skew the result. This is known as attrition bias. Say, for example, many older patients drop out of the treatment group due to catching the flu, leaving the distribution of the treatment group skewed younger and healthier. If the data analysts don’t take this skewed attrition into account, they may find the treatment correlated with some surprising health (and youth)-giving effects.
Interpretation Bias. Due to the nature of the exercise, clinical trial design requires both medical and statistical expertise, and misuse of statistical analysis is rife. For a number of reasons, trials may be conducted with a number of patients(referred to in summaries as n) insufficient to provide statistically significant data. Additionally, if you consider the ideal sample size to be every living human being on the planet, a small group increases the likelihood of the selection of a biased group that by chance over-represents some less-than-ubiquitous trait among the population.
Methodology. The test methodology itself is, of course, hugely important, perhaps especially so in the study of nootropics, as the tests themselves are objective measures of subjects ability to perform tasks that are meant to generalize to quantifiable aspects of mental performance. More on test design later in this series, but for this part we should simply say that the conclusions drawn from a study of the sort that are typically done on nootropics should be very narrowly interpreted, and extrapolations from those conclusions regarded skeptically at best. It’s one thing to demonstrate that a process synthesizes a chemical, but when you’re dealing with the human mind, objectivity itself a bit more elusive, and tests are likely to be used as proxies for a broader idea about consciousness.
External validity is the term for the extent to which the results of trials provide the correct basis for generalization to other circumstances. In our case, we want to be thinking about whether or not we can apply the findings of a study to ourselves. Let’s say you’re a healthy young person, and the study in question was done on older adults with early-onset Alzheimers. Do the results of the study necessarily apply to you? If they’re about wearing a knee brace when playing tennis, perhaps. But if they’re about the use of a nootropic? Possibly not.
Early-Stage Research. This one is easy to look out for — many published studies are done in vitro (‘in a petri dish’), some are done in vivo (‘in a living organism’), which sounds great, but that organism might be a mouse, or just cells, rather than humans. Neither these sorts of studies produces conclusions that you should base your supplement decisions on, because you are neither a cell culture nor a mouse. Unless, of course, they find that ‘x treatment kills everything it touches’, in which case, that’s probably actionable information.

Fun editorial note here — my science advisors informed me that I have touched upon a minor hot-button issue here with my framing of the ‘in vitro’ vs ‘in vivo’ distinction as it applies to cells. I am not taking sides because, despite 6 years of Latin studies, I am unqualified.

Conflicts of Interest. Another easy one to spot. This topic is challenging to generalize on, though, as simply being funded by the maker of the treatment being tested doesn’t necessarily invalidate or bias the conclusion of the study, and if we threw out every study that was paid for by the maker of the thing being tested, we’d end up with few medicines or medical devices at all. It’s just something to be aware of when looking at the study. More on this later.
Publication Bias. Related to the point about conflict of interest above — this last one is something you can do nothing about, but still bears mentioning — despite that researchers can volunteer to commit to do so, in the US, you are under no obligation to publish the results of a trial if they do not comport with the outcome you were hoping to find. This is called publication bias, and it’s not limited to medical research, but its impact cannot be overstated in skewing public perception and scientific consensus. There are loads of studies that, because they found the opposite outcome the researchers hypothesized, or found no conclusion at all, were never published, even though they were well-designed and yielded actionable information.
Etc, etc. As mentioned above, this list could go on forever, and the medical community is constantly refining what are considered best practices for each stage of a trial’s design.

The Takeaways:

1) It’s very difficult to summarize the findings of a study without over-simplifying and leaving out important information necessary to understanding the value and applicability of the result.

2) It’s very difficult to design a good study, period.

3) Doctors are trained to read studies and even they argue about how to interpret the results.

Where does that leave us when trying to make informed decisions? We’re shopping for nootropics already — now we have to go to medical school to figure out what to take?

It’s like you need to be taking brain pills to understand nootropics

The Outro

This concludes part 2 of Nootropics for Dummies, because I’m gonna keep these bite-sized. Not because you’re a dummy — I just know you’re busy.

Read on for Part 3: EVIDENCE, Part 2, where we’ll look at where to get good evidence, and, perhaps better yet, where to find meta-analysis and criticism of said evidence, and follow me to be notified when I post more.

Give me an example / Just tell me what to take / The Shortcut

I get it. With the trials and the bias and so on and you were here because you’re overwhelmed already. I felt the same way at one point, and I’m flattered you already want my opinion.

I suppose it won’t be a spoiler to say that I figured out something that works for me, and that, nearly 3 years later, it’s now available for anybody, whether or not they want to read the rest of this article. It’s called Plato, it’s exactly what I’ve been taking for 2.5 years†, and you can hit that link to get it.

† I’m not Rain Man now but more on that soon.††

†† More like The Lawnmower Man, a recurring joke I think you’ll continue to enjoy throughout this series.