Will Fortune Favor the BOLD?

by Leyla Loued-Khenissi

Controversy in scientific research is a necessary feature of scientific advancement. Less common are controversies that spill out into the public domain, replete with glib, unconditional headlines that are neither written by scientists nor intended for a discerning public. In early 2016, such a controversy boiled over from a staid scientific journal to general media outlets. The story stemmed from a paper in PNAS that scrutinized commonly used statistical methods in functional magnetic resonance imaging (fMRI). that yield an increased rate of false positives. fMRI is a method based on magnetic resonance that measures blood oxygenation (BOLD) levels in the brain with the use of a very powerful magnet. The BOLD signal is a proxy for brain activity — the more oxygen is recruited, the more the brain is working. This PNAS paper promptly bolstered many scientists’ suspicions about fMRI. The findings therein provided a necessary and valuable critique of MRI analysis methods and was thus an objective plus to the field, but the most galvanizing result was a collective meltdown in the neuroimaging community over our particular method of choice. It was the zombie fish scan(dal) all over again, albeit with better statistics and without the comedic element to soften the blow.

Thankfully, since (and even prior) to the PNAS article, several suggestions have been outlined to address fMRI’s shortcomings. Notably, the Organization for Human Brain Mapping (OHBM) committee advised fMRI researchers to ‘keep calm and scan on’, providing a much needed dampening to the unbridled alarm the PNAS article triggered.

fMRI studies suffer from small sample sizes and costs associated with scanning time in addition to limited access and time constraints, as well as, in some cases, limited recruitment possibilities from a given population, all of which currently conspire to keep sample sizes low relative to what is needed to measure a meaningful effect size. One solution to low statistical power, which has been mentioned before, is to pool resources. Given limited resources, sharing has become part of the zeitgeist in other aspects of society, but it would seem especially well suited to scientific efforts, given their sources of funding (public) and universal mandate (collective progress). That is, we ought to share data while we wait for costs to go down and access to grow; costs will go down as time goes on. Not so long ago, data storage alone was a headache whereas now it is just a nuisance. Thankfully, the state of technology is like Southern California weather; if you don’t like it right now, just wait a minute.

While discussion has been rife within the neuroimaging community since the findings of the PNAS article rippled out to the general public, we must consider a loss of confidence in the fMRI method that cannot be explained away with more neuroimaging jargon. The imaging community’s response to colleagues that employ other methods, as well as to the layperson, has been muted though no less necessary. We must address the doubt that exists beyond neuroimagers to avoid needlessly relegating this incredible tool to the junkyard heap of pseudoscientific methods. A first point to outline is that, plausible deniability notwithstanding, fMRI is, as far as scientific methods go, a sexy method with an even sexier output. For years it has been emblematic of cutting edge technology and the potential of the future. Slap a high- or even low-resolution image of a brain scan on a product or promotional item and one’s tech cred (at least relative to non brain researchers) soars. Few technological advances have elicited such unabashed wonder in the eyes of the general public, who are often excluded from the less visually accessible wonders of scientific advances; nobody needs a degree to understand what a lit-up brain implies. Researchers that employ other methods, which yield just as galvanizing results albeit in a less visually appealing form, may have felt a nagging feeling that fMRI was unfairly stealing their thunder. Thus, just like the beautiful starlet, MRI has always been regarded with suspicion and its shortcomings more readily vulnerable to scrutiny. In a first instance then, it is of no surprise that the method’s (deserved) criticisms be highlighted as they have been.

Figure 1: What would you rather? a) Tyrosine Hydroxylase (TH)+ cells in rat Ventral Tegmental Area (Courtesy I. Zalachoras, Laboratory of Behavioral Genetics, EPFL). b) EEG recording in the rat (Courtesy S. Astori, Laboratory of Behavioral Genetics, EPFL) c) 36 year old male listening to instructions, fMRI (FWE = .05) (L. Loued-Khenissi, EPFL).

That MRI is less valid or reliable than other tools is, however, up for debate. Most other tools of the trade measure indirect indices, assess a quantity by its proxy, investigate a hidden phenomenon by its outward observation, and all this with various sources of noise and error. Therefore we must be cautious in castigating MRI as being especially suspect relative to other methods.

In support of the continued use and confidence in MRI is the reminder that most neuro-imagers have been aware of the statistical liabilities related to the method and have been working to address known sources of statistical error with the development of different a priori assumptions on the brain and MR technology, as well as the application of different tests to account for trade-offs in assessing imaging output. The degree of failure uncovered highlights the increased need for caution, more data, more extensive consultation with other sub-fields of neuroscience, and further development of acquisition and analytical tools. This last route in particular provides us with an opportunity to go beyond protocol in our methodology and toy with creative solutions. Using new tools or introducing new assumptions necessarily introduces new potential sources of failure, but this is a cost that must be assumed for the sake of progress. Interestingly, the opposite has been offered as a solution as well: that is, fMRI researchers should use set, standardized methods for analyses. However, that relegates scientists to the safety zone, or to the role of technician as defined by Carl Sagan.

Another general point to make about fMRI or any other scientific endeavor is that it is difficult, it is iterative, and thus failure should be expected. All these features relate to the novelty inherent to research. The hunch and some luck can go a way towards minimizing the time it takes to answer a research question, but deadlines do not serve scientific success. Because we cannot force luck or accelerate the research process to meet contractual, academic, professional or funding deadlines, the pressure to cut corners exists. Papers and results are published, as the profession requires, but the demand for quantity outstrips their quality. This point has also been raised in the past. Here’s a thought experiment: what if there were no time constraints on research? What if, upon obtaining degrees, scientists could simply work for a salary and be provided with operating costs to perform research during their professional lives? The point above touches on a potential shift in how we view work, time and compensation in general — but that particular argument is beyond the scope of this piece. What we can call on here is for senior researchers to alter the hierarchical system of research and consider that raising the bar for scientific quality while not lowering it for quantity is unsustainable.

In the end, fMRI is an amazing piece of technology. A painless, non-invasive method capable of tracking (albeit with some drag) neural activity on and in human brains, while the latter are awake and well, with near immediate results. Hopefully, it will be replaced with faster, better, cheaper technology, in the near future but for now the method is peerless. The onus falls on 1) fMRI researchers to challenge themselves on how best to deploy it; to refrain from viewing it as a few-click solution and accept that there is no free lunch — we have to keep developing tools to increase signal-to-noise, control for artifacts, lower false positives. Our tasks, assumptions and analytical designs must be prepared with more thought, time and input from other research areas, and we must share data to output high quality results. 2) We must simultaneously seek solidarity within the wider community — first from our very own senior researchers, the latter whom may admonish junior scientists on data analysis methods today, but have perhaps benefitted from formerly lax thresholds for publication and who currently have more scientific capital than younger scientists to challenge the status quo. 3) We must also entreat funding agencies to provide us with more time: behavioral scientists are aware of a speed-accuracy trade-off in tasks — why would it be different in research? 4) We can and should challenge the current paradigm of science as an individual competitive sport; research questions abound, especially in the neurosciences, thus there is plenty of work for generations of inquiring minds. 5) Finally, accepting that any method of choice in neuroscientific study is just a tool would go a long way in untangling our willful misunderstanding of each other’s languages. In the end we are all studying brains and their functions; to remain monolingual, so-to-speak, is a parochial pigeonholing whose end result is creative poverty. Making use of findings obtained in other fields — not just in a cursory manner to support our data but also with the aim of sharpening our initial hypotheses — would contribute to the field as well as support a sense of inclusiveness necessary for the field to advance.

Leyla Loued-Khenissi is a doctoral student at EPFL’s Brain Mind Institute in Lausanne, Switzerland, and the winner of the Science Writing Competiton at the 2016 Human Brain Project Summit. Her main areas of interest are psychophysics, neuroscience and neuroeconomics.