What Machine Learning Scientist/Engineer Should be Aware of before Getting into a Brain Imaging Project

As a machine learning (ML) or data scientist, your goal is often to evaluate business or research questions when looking at new ideas before getting deep. Brain imaging (BI), especially functional one, is widely adopted to study brain states under various conditions or diseases. It is often used for the prognosis and diagnosis of many brain disorders. Often ML problem is formulated as regression/classification or segmentation from spatio/temporal data but under some considerations.

The end goal of BI research is to translate research and findings into therapeutic products. This translation can be an uphill battle for functional BI due to many reasons [1]. This might seem odd with all the advancements in life science fields. I will try to lay out several observations from working in neuroscience.

First, let us understand BI data from the ML side.

Brain Imaging

BI is a set of techniques to measure brain states and properties. (BI) can be divided into structural and functional imaging. Structural imaging measures static properties of the brain like cortical thickness, volumes of brain regions structural connectivity among brain regions. There are several prominent approaches in structural imaging, including Magnetic resonance imaging (MRI), Diffusion Tensor Imaging (DTI), and Computed Tomography (CT). Typically, such data are represented by 3D images/matrices or sometimes sparse 3D matrices. ML is concerned with performing classification/regression or segmentation from data (e.g., predicting Alzheimer’s disease from an MRI scan or tumor segmentation from a CT scan). On the other hand, functional BI is concerned with measuring brain state (activation) over time. Examples of functional BI include Magnetoencephalography (MEG), electroencephalogram (EEG), Positron emission tomography (PET), and functional MRI (fMRI). Data is usually presented as 4D matrices for fMRI and PET ( 3D brain x 1D time) or 2D (number of channels x 1D time) for EEG and MEG. In some cases, BI data are represented as tabular data after extracting summary information from processed data. For example, structural data can be further summarized as average volumes after parsing the brain into regions using brain atlases. fMRI data can be represented as pairwise correlations (functional connectivity) among brain regions.

BI data acquisition and processing are very heavy signal processing tasks and require significant domain knowledge. A couple of points can be noticed when working with BI, especially functional BI.

Four important points that every ML scientist/engineer should be aware of:

The Availability and Accessibility of Data

Benchmark datasets (e.g., MNIST, CelebA, LM1B, etc.) are pillars for developing and advancing ML research. The availability of data in the life sciences domain is a general challenge for ML research. However, the issue is well pronounced in BI, especially on the functional side. Most of the datasets in BI have a limited number of samples (in best cases couple of thousands), and most of the large-scale BI data cohorts like UK Biobank (50K-100K) [2], ENIGMA [3], and ABCD [4] require certain academic affiliation in addition to significant fees (e.g., UK Biobank). In comparison, many genomics data are freely available through some large repositories like ENCODE [5], GEO[6], 4DN [7], and so on. I should say the scarcity of the data in the BI domain comes from different reasons related to the high cost of data collection and the identifiability of participants. But, part of the issue with BI comes from researchers who sometimes don’t like to share data. It was not until recently that there has been an effort to gather and organize published work data in one place (e.g., ENIGMA).

The Nature of Questions

One fundamental challenge with BI is whether ML is able to answer the research question or not. When using ML to solve BI problems, there is an assumption that BI data carry a signal related to disease or conditions. This might be true for some brain disorders that carry out significant structural and functional alterations to the brain, like brain cancer, Alzheimer’s, and Parkinson’s disease. However, for many diseases that don’t have a significant impact, detecting disease signals might be challenging, especially using functional BI. This is obvious if you think about how different our brains are. It would be pretty challenging to make a diagnosis (pattern recognition) by scanning someone’s brain to claim a diagnosis unless symptoms are dominant.

Additionally, many brain disorders are expressed as changes at the level of molecules, cells, and tissues, which are then manifested with different alteration paths leading to structural and functional changes in the brain. Therefore, it might be challenging for ML to detect disease signals from BI data unless augmented with prior biological information. To add more complexity to ML problems in BI, some psychiatric disorders can manifest themselves in different ways based on each participant (e.g., anxiety and depression).

Challenges and Reproducibility of BI data

BI as a technology is still new and requires a significant setup. Data has to go through dozens of preprocessing steps. Moreover, data are very sensitive to scanner calibrations and hardware configurations. One daily issue is scanner drifts that often need to be accounted for. Thus, from the perspective of ML, this imposes a limitation since ML, in many cases, relies on already processed data; thus, it is difficult to build general models as in image classification or natural language processing (NLP). Furthermore, no one can argue about the reproducibility crisis in the functional BI domain [8, 9]. Due to the low sample size and variations in preprocessing pipelines, it was often difficult to reproduce many imaging studies, which adds more challenges to producing robust ML models.

Availability of Public BI Models

One can notice that there are not many benchmarks to validate or compare different ML models or even to make domain-specific transfer learning in BI. ML scientists will often have to trust current models they have and optimize upon them rather than optimize upon other models. Sharing models will at least help BI to advance further. BI can learn very good lessons from AlphaFold2; After DeepMind published its model, there has been a tremendous amount of progress in protein structure prediction and even reproduction of similar algorithms like OpenFold and RoseTTAFold. Similarly, ML-genomic domains have done a great job in offering many models to the public.

Here are five questions that should be asked by ML engineers before running BI projects.

Questions that every ML scientist should ask before running the BI project

1. How many data points are in the study, and what is the data type? It is an obvious question, but given the complexity of BI data and potential variations, it should be emphasized from the beginning.

Scenario: A study with n fMRI participants with the goal from ML to classify participants into healthy and non-healthy. ML scientists should ask how many participants are in the study. Will the ML development run on summary statistics of the processed data (tabular data) or processed BI data (3D or 4D). The latter requires a significant number of samples [thousands] [8].

2. Did all data come from the same scanner/hardware? Did all data go through the same preprocessing pipeline?

Scenario: A collaboration study with n EEG participants gathered from three labs over two years (potential variation in preprocessing pipeline + hardware drift).

3. What is the effect of covariates?

Scenario: An fMRI study with a wide range of ages across groups. ML scientists should be aware that age, sex, education, etc. have a significant effect on BI data and should plan to design a rigorous validation [e.g., stratified cross-validation]).

4. Are there any hidden covariates? (require domain-expert involvement).

Scenario: An fMRI study with a good sample size to differentiate participants with anxiety from healthy ones. Anxious participants tend to move a lot in scanners introducing so many artifacts and changes in the signal compared to healthy participants. Thus, an ML algorithm might pick that; the ML model might learn how to differentiate between calm and moving participants but not learn anxious brain or not. ML scientists should probe the hidden covariates.

5. Is the initial analysis of the data in line with the literature?

Scenario: An EEG study with a good number of samples shows high alpha activity in a particular brain region, which is inconsistent with the literature. ML scientists should ask if the initial analysis is in line with the literature, which serves as a sanity check. Typically, before moving far with ML, the ground truth should be established ). Through model explainability tools like SHAP [10] or LIME [11], ML scientists should confirm findings from the literature.

Conclusion

I shed light on some of the challenges that face ML applications in BI and have to do with the scarcity of data, the complexity of BI, and the lack of data sharing. I highlighted some of the issues that ML researchers might face when handling BI data, and a set of questions should be asked early before going deep into BI projects. Although I did not provide solutions for each of the five questions, these questions will trigger ML scientists to decide on algorithms and validation strategies. While some challenges are difficult to address in BI, large-scale data initiatives and broader data accessibility might help to mitigate obstacles in BI.

Author

Obada Al Zoubi, Ph.D. A biomedical data and machine learning scientist interested in solving machine learning problems in genomics, neuroimaging and signal processing fields.

References

[1] “Revisiting doubt in neuroimaging research,” Nature Neuroscience, 2022/07/05, 2022.

[2] C. Sudlow, J. Gallacher, N. Allen, V. Beral, P. Burton, J. Danesh, P. Downey, P. Elliott, J. Green, and M. Landray, “UK biobank: an open access resource for identifying the causes of a wide range of complex diseases of middle and old age,” PLoS medicine, vol. 12, no. 3, pp. e1001779, 2015.

[3] P. M. Thompson, N. Jahanshad, C. R. Ching, L. E. Salminen, S. I. Thomopoulos, J. Bright, B. T. Baune, S. Bertolín, J. Bralten, and W. B. Bruin, “ENIGMA and global neuroscience: A decade of large-scale studies of the brain in health and disease across more than 40 countries,” Translational psychiatry, vol. 10, no. 1, pp. 1–28, 2020.

[4] B. J. Casey, T. Cannonier, M. I. Conley, A. O. Cohen, D. M. Barch, M. M. Heitzeg, M. E. Soules, T. Teslovich, D. V. Dellarco, and H. Garavan, “The adolescent brain cognitive development (ABCD) study: imaging acquisition across 21 sites,” Developmental cognitive neuroscience, vol. 32, pp. 43–54, 2018.

[5] J. R. Ecker, W. A. Bickmore, I. Barroso, J. K. Pritchard, Y. Gilad, and E. Segal, “ENCODE explained,” Nature, vol. 489, no. 7414, pp. 52–54, 2012.

[6] T. Barrett, S. E. Wilhite, P. Ledoux, C. Evangelista, I. F. Kim, M. Tomashevsky, K. A. Marshall, K. H. Phillippy, P. M. Sherman, and M. Holko, “NCBI GEO: archive for functional genomics data sets — update,” Nucleic acids research, vol. 41, no. D1, pp. D991-D995, 2012.

[7] S. B. Reiff, A. J. Schroeder, K. Kırlı, A. Cosolo, C. Bakker, S. Lee, A. D. Veit, A. K. Balashov, C. Vitzthum, and W. Ronchetti, “The 4D Nucleome Data Portal as a resource for searching and visualizing curated nucleomics data,” Nature communications, vol. 13, no. 1, pp. 1–11, 2022.

[8] S. Marek, B. Tervo-Clemmens, F. J. Calabro, D. F. Montez, B. P. Kay, A. S. Hatoum, M. R. Donohue, W. Foran, R. L. Miller, and T. J. Hendrickson, “Reproducible brain-wide association studies require thousands of individuals,” Nature, vol. 603, no. 7902, pp. 654–660, 2022.

[9] R. E. Kelly Jr, and M. J. Hoptman, “Replicability in Brain Imaging,” 3, MDPI, 2022, p. 397.

[10] S. M. Lundberg, and S.-I. Lee, “A unified approach to interpreting model predictions,” Advances in neural information processing systems, vol. 30, 2017.

[11] M. T. Ribeiro, S. Singh, and C. Guestrin, “” Why should i trust you?” Explaining the predictions of any classifier.” pp. 1135–1144.