What Is the Test-Retest Reliability of Common Task-Functional MRI Measures? New Empirical Evidence and a Meta-Analysis

Overview

Journal Psychol Sci

Specialty Psychology

Date 2020 Jun 4

PMID 32489141

Citations 301

Authors

Maxwell L Elliott

Annchen R Knodt

David Ireland

Meriwether L Morris

Richie Poulton

Sandhya Ramrakha

Maria L Sison

Terrie E Moffitt

Avshalom Caspi

Ahmad R Hariri

Affiliations

Soon will be listed here.

Abstract

Identifying brain biomarkers of disease risk is a growing priority in neuroscience. The ability to identify meaningful biomarkers is limited by measurement reliability; unreliable measures are unsuitable for predicting clinical outcomes. Measuring brain activity using task functional MRI (fMRI) is a major focus of biomarker development; however, the reliability of task fMRI has not been systematically evaluated. We present converging evidence demonstrating poor reliability of task-fMRI measures. First, a meta-analysis of 90 experiments ( = 1,008) revealed poor overall reliability-mean intraclass correlation coefficient (ICC) = .397. Second, the test-retest reliabilities of activity in a priori regions of interest across 11 common fMRI tasks collected by the Human Connectome Project ( = 45) and the Dunedin Study ( = 20) were poor (ICCs = .067-.485). Collectively, these findings demonstrate that common task-fMRI measures are not currently suitable for brain biomarker discovery or for individual-differences research. We review how this state of affairs came to be and highlight avenues for improving task-fMRI reliability.

Citing Articles

A systematic review of structural neuroimaging markers of psychotherapeutic and pharmacological treatment for obsessive-compulsive disorder.

Moreau A, Hansen I, Bogdan R Front Psychiatry. 2025; 15:1432253.

PMID: 40018086 PMC: 11865061. DOI: 10.3389/fpsyt.2024.1432253.

Neural correlates and plasticity of explicit emotion regulation following the experience of trauma.

Konrad A, Miu A, Trautmann S, Kanske P Front Behav Neurosci. 2025; 19:1523035.

PMID: 40017732 PMC: 11865028. DOI: 10.3389/fnbeh.2025.1523035.

Longitudinal analysis of the ABCD® study.

Hawes S, Littlefield A, Lopez D, Sher K, Thompson E, Gonzalez R Dev Cogn Neurosci. 2025; 72:101518.

PMID: 39999579 PMC: 11903845. DOI: 10.1016/j.dcn.2025.101518.

Hippocampal network connectivity and episodic memory in individuals aging with traumatic brain injury.

Carpenter C, Mullin H, Cwiek A, Carter E, Vervoordt S, Lan X Brain Imaging Behav. 2025; .

PMID: 39982608 DOI: 10.1007/s11682-025-00979-x.

Multivariate Neural Markers of Individual Differences in Thought Control Difficulties.

DeRosa J, Smolker H, Kim H, Groff B, Peacock J, Lewis-Peacock J bioRxiv. 2025; .

PMID: 39975087 PMC: 11838559. DOI: 10.1101/2025.02.04.636283.

References

Woo C, Chang L, Lindquist M, Wager T . Building better biomarkers: brain models in translational neuroimaging. Nat Neurosci. 2017; 20(3):365-377. PMC: 5988350. DOI: 10.1038/nn.4478. View

Kriegeskorte N, Lindquist M, Nichols T, Poldrack R, Vul E . Everything you never wanted to know about circular analysis, but were afraid to ask. J Cereb Blood Flow Metab. 2010; 30(9):1551-7. PMC: 2949251. DOI: 10.1038/jcbfm.2010.86. View

Button K, Ioannidis J, Mokrysz C, Nosek B, Flint J, Robinson E . Power failure: why small sample size undermines the reliability of neuroscience. Nat Rev Neurosci. 2013; 14(5):365-76. DOI: 10.1038/nrn3475. View

Hajcak G, Meyer A, Kotov R . Psychometrics and the neuroscience of individual differences: Internal consistency limits between-subjects effects. J Abnorm Psychol. 2017; 126(6):823-834. DOI: 10.1037/abn0000274. View

Vanderwal T, Eilbott J, Castellanos F . Movies in the magnet: Naturalistic paradigms in developmental functional neuroimaging. Dev Cogn Neurosci. 2018; 36:100600. PMC: 6969259. DOI: 10.1016/j.dcn.2018.10.004. View

Nosek B, Ebersole C, DeHaven A, Mellor D . The preregistration revolution. Proc Natl Acad Sci U S A. 2018; 115(11):2600-2606. PMC: 5856500. DOI: 10.1073/pnas.1708274114. View

Van Essen D, Smith S, Barch D, Behrens T, Yacoub E, Ugurbil K . The WU-Minn Human Connectome Project: an overview. Neuroimage. 2013; 80:62-79. PMC: 3724347. DOI: 10.1016/j.neuroimage.2013.05.041. View

Egger M, Davey Smith G, Schneider M, Minder C . Bias in meta-analysis detected by a simple, graphical test. BMJ. 1997; 315(7109):629-34. PMC: 2127453. DOI: 10.1136/bmj.315.7109.629. View

Gratton C, Laumann T, Nielsen A, Greene D, Gordon E, Gilmore A . Functional Brain Networks Are Dominated by Stable Group and Individual Factors, Not Cognitive or Daily Variation. Neuron. 2018; 98(2):439-452.e5. PMC: 5912345. DOI: 10.1016/j.neuron.2018.03.035. View

10.

Han X, Jovicich J, Salat D, van der Kouwe A, Quinn B, Czanner S . Reliability of MRI-derived measurements of human cerebral cortical thickness: the effects of field strength, scanner upgrade and manufacturer. Neuroimage. 2006; 32(1):180-94. DOI: 10.1016/j.neuroimage.2006.02.051. View

11.

Chen G, Taylor P, Haller S, Kircanski K, Stoddard J, Pine D . Intraclass correlation: Improved modeling approaches and applications for neuroimaging. Hum Brain Mapp. 2017; 39(3):1187-1206. PMC: 5807222. DOI: 10.1002/hbm.23909. View

12.

Swartz J, Knodt A, Radtke S, Hariri A . A neural biomarker of psychological vulnerability to future life stress. Neuron. 2015; 85(3):505-11. PMC: 4319095. DOI: 10.1016/j.neuron.2014.12.055. View

13.

Schafer T, Schwarz M . The Meaningfulness of Effect Sizes in Psychological Research: Differences Between Sub-Disciplines and the Impact of Potential Biases. Front Psychol. 2019; 10:813. PMC: 6470248. DOI: 10.3389/fpsyg.2019.00813. View

14.

Elliott M, Belsky D, Anderson K, Corcoran D, Ge T, Knodt A . A Polygenic Score for Higher Educational Attainment is Associated with Larger Brains. Cereb Cortex. 2018; 29(8):3496-3504. PMC: 6645179. DOI: 10.1093/cercor/bhy219. View

15.

Nord C, Gray A, Charpentier C, Robinson O, Roiser J . Unreliability of putative fMRI biomarkers during emotional face processing. Neuroimage. 2017; 156:119-127. PMC: 5553850. DOI: 10.1016/j.neuroimage.2017.05.024. View

16.

Poulton R, Moffitt T, Silva P . The Dunedin Multidisciplinary Health and Development Study: overview of the first 40 years, with an eye to the future. Soc Psychiatry Psychiatr Epidemiol. 2015; 50(5):679-93. PMC: 4412685. DOI: 10.1007/s00127-015-1048-8. View

17.

Cicchetti D, Sparrow S . Developing criteria for establishing interrater reliability of specific items: applications to assessment of adaptive behavior. Am J Ment Defic. 1981; 86(2):127-37. View

18.

Yarkoni T, Westfall J . Choosing Prediction Over Explanation in Psychology: Lessons From Machine Learning. Perspect Psychol Sci. 2017; 12(6):1100-1122. PMC: 6603289. DOI: 10.1177/1745691617693393. View

19.

DuBois J, Adolphs R . Building a Science of Individual Differences from fMRI. Trends Cogn Sci. 2016; 20(6):425-443. PMC: 4886721. DOI: 10.1016/j.tics.2016.03.014. View

20.

Hedge C, Powell G, Sumner P . The reliability paradox: Why robust cognitive tasks do not produce reliable individual differences. Behav Res Methods. 2017; 50(3):1166-1186. PMC: 5990556. DOI: 10.3758/s13428-017-0935-1. View