Impact of ERP Reliability Cutoffs on Sample Characteristics and Effect Sizes: Performance-Monitoring ERPs in Psychosis and Healthy Controls

Overview

Journal Psychophysiology

Specialty Psychiatry

Date 2025 Feb 17

PMID 39957549

Authors

Gavin Heindorf

Amanda Holbrook

Bohyun Park

Gregory A Light

Philippe Rast

Dan Foti

Roman Kotov

Peter E Clayson

Affiliations

Soon will be listed here.

Abstract

In studies of event-related brain potentials (ERPs), it is common practice to exclude participants for having too few trials for analysis to ensure adequate score reliability (i.e., internal consistency). However, in research involving clinical samples, the impact of increasingly rigorous reliability standards on factors such as sample generalizability, patient versus control effect sizes, and effect sizes for within-group correlations with external variables is unclear. This study systematically evaluated whether different ERP reliability cutoffs impacted these factors in psychosis. Error-related negativity (ERN) and error positivity (Pe) were assessed during a modified flanker task in 97 patients with psychosis and 104 healthy comparison participants, who also completed measures of cognition and psychiatric symptoms. ERP reliability cutoffs had notably different effects on the factors considered. A recommended reliability cutoff of 0.80 resulted in sample bias due to systematic exclusion of patients with relatively few task errors, lower reported psychiatric symptoms, and higher levels of cognitive functioning. ERP score reliability lower than 0.80 resulted in generally smaller between- and within-group effect sizes, likely misrepresenting effect sizes. Imposing rigorous ERP reliability standards in studies of psychotic disorders might exclude high-functioning patients, which raises important considerations for the generalizability of clinical ERP research. Moving forward, we recommend examining characteristics of excluded participants, optimizing paradigms and processing pipelines for use in clinical samples, justifying reliability thresholds, and routinely reporting score reliability of all measurements, ERP or otherwise, used to examine individual differences, especially in clinical research.

References

Baldwin S, Larson M, Clayson P . The dependability of electrophysiological measurements of performance monitoring in a clinical sample: A generalizability and decision analysis of the ERN and Pe. Psychophysiology. 2015; 52(6):790-800. DOI: 10.1111/psyp.12401. View

Hedge C, Powell G, Sumner P . The reliability paradox: Why robust cognitive tasks do not produce reliable individual differences. Behav Res Methods. 2017; 50(3):1166-1186. PMC: 5990556. DOI: 10.3758/s13428-017-0935-1. View

Nosek B, Ebersole C, DeHaven A, Mellor D . The preregistration revolution. Proc Natl Acad Sci U S A. 2018; 115(11):2600-2606. PMC: 5856500. DOI: 10.1073/pnas.1708274114. View

Brand A, Bradley M . The Precision of Effect Size Estimation From Published Psychological Research: Surveying Confidence Intervals. Psychol Rep. 2018; 118(1):154-170. DOI: 10.1177/0033294115625265. View

Flegal K, Kit B, Graubard B . Bias in Hazard Ratios Arising From Misclassification According to Self-Reported Weight and Height in Observational Studies of Body Mass Index and Mortality. Am J Epidemiol. 2018; 187(1):125-134. PMC: 5859975. DOI: 10.1093/aje/kwx193. View

Thigpen N, Kappenman E, Keil A . Assessing the internal consistency of the event-related potential: An example analysis. Psychophysiology. 2016; 54(1):123-138. PMC: 5525326. DOI: 10.1111/psyp.12629. View

L Seghier M, Price C . Interpreting and Utilising Intersubject Variability in Brain Function. Trends Cogn Sci. 2018; 22(6):517-530. PMC: 5962820. DOI: 10.1016/j.tics.2018.03.003. View

Barch D, Braver T, Carter C, Poldrack R, Robbins T . CNTRICS final task selection: executive control. Schizophr Bull. 2008; 35(1):115-35. PMC: 2643948. DOI: 10.1093/schbul/sbn154. View

Larson M, Clayson P, Clawson A . Making sense of all the conflict: a theoretical review and critique of conflict-related ERPs. Int J Psychophysiol. 2014; 93(3):283-97. DOI: 10.1016/j.ijpsycho.2014.06.007. View

10.

Falkenstein M, Hohnsbein J, Hoormann J, Blanke L . Effects of crossmodal divided attention on late ERP components. II. Error processing in choice reaction tasks. Electroencephalogr Clin Neurophysiol. 1991; 78(6):447-55. DOI: 10.1016/0013-4694(91)90062-9. View

11.

Clayson P, Carbine K, Baldwin S, Olsen J, Larson M . Using generalizability theory and the ERP Reliability Analysis (ERA) Toolbox for assessing test-retest reliability of ERP scores part 1: Algorithms, framework, and implementation. Int J Psychophysiol. 2021; 166:174-187. DOI: 10.1016/j.ijpsycho.2021.01.006. View

12.

Charles L, Gaillard R, Amado I, Krebs M, Bendjemaa N, Dehaene S . Conscious and unconscious performance monitoring: Evidence from patients with schizophrenia. Neuroimage. 2016; 144(Pt A):153-163. DOI: 10.1016/j.neuroimage.2016.09.056. View

13.

Klingelhofer-Jens M, Ehlers M, Kuhn M, Keyaniyan V, Lonsdorf T . Robust group- but limited individual-level (longitudinal) reliability and insights into cross-phases response prediction of conditioned fear. Elife. 2022; 11. PMC: 9691022. DOI: 10.7554/eLife.78717. View

14.

Foti D, Kotov R, Hajcak G . Psychometric considerations in using error-related brain activity as a biomarker in psychotic disorders. J Abnorm Psychol. 2013; 122(2):520-531. DOI: 10.1037/a0032618. View

15.

Kolossa A, Kopp B . Data quality over data quantity in computational cognitive neuroscience. Neuroimage. 2018; 172:775-785. DOI: 10.1016/j.neuroimage.2018.01.005. View

16.

Shafer A . Meta-analysis of the brief psychiatric rating scale factor structure. Psychol Assess. 2005; 17(3):324-35. DOI: 10.1037/1040-3590.17.3.324. View

17.

Baldwin S . Improving the rigor of psychophysiology research. Int J Psychophysiol. 2016; 111:5-16. DOI: 10.1016/j.ijpsycho.2016.04.006. View

18.

Kozak M, Cuthbert B . The NIMH Research Domain Criteria Initiative: Background, Issues, and Pragmatics. Psychophysiology. 2016; 53(3):286-97. DOI: 10.1111/psyp.12518. View

19.

Volpert-Esmond H . Looking at Change: Examining Meaningful Variability in Psychophysiological Measurements. Biol Psychiatry Cogn Neurosci Neuroimaging. 2022; 7(6):530-531. DOI: 10.1016/j.bpsc.2022.02.006. View

20.

Walters R, Hoffman L, Templin J . The Power to Detect and Predict Individual Differences in Intra-Individual Variability Using the Mixed-Effects Location-Scale Model. Multivariate Behav Res. 2018; 53(3):360-374. DOI: 10.1080/00273171.2018.1449628. View