Protecting Against Researcher Bias in Secondary Data Analysis: Challenges and Potential Solutions

Overview

Journal Eur J Epidemiol

Specialty Public Health

Date 2022 Jan 13

PMID 35025022

Authors

Jessie R Baldwin

Jean-Baptiste Pingault

Tabea Schoeler

Hannah M Sallis

Marcus R Munafo

Affiliations

Soon will be listed here.

Abstract

Analysis of secondary data sources (such as cohort studies, survey data, and administrative records) has the potential to provide answers to science and society's most pressing questions. However, researcher biases can lead to questionable research practices in secondary data analysis, which can distort the evidence base. While pre-registration can help to protect against researcher biases, it presents challenges for secondary data analysis. In this article, we describe these challenges and propose novel solutions and alternative approaches. Proposed solutions include approaches to (1) address bias linked to prior knowledge of the data, (2) enable pre-registration of non-hypothesis-driven research, (3) help ensure that pre-registered analyses will be appropriate for the data, and (4) address difficulties arising from reduced analytic flexibility in pre-registration. For each solution, we provide guidance on implementation for researchers and data guardians. The adoption of these practices can help to protect against researcher bias in secondary data analysis, to improve the robustness of research based on existing data.

Citing Articles

Exploring the Intersection of Mental and Reproductive Health Among Women Living with HIV in Spain: A Qualitative Secondary Data Analysis.

Huertas-Zurriaga A, Gimenez-Diez D, Leyva-Moral J Healthcare (Basel). 2025; 13(2).

PMID: 39857194 PMC: 11764562. DOI: 10.3390/healthcare13020168.

A guide for planning triangulation studies to investigate complex causal questions in behavioural and psychiatric research.

Treur J, Lukas E, Sallis H, Wootton R Epidemiol Psychiatr Sci. 2024; 33:e61.

PMID: 39506622 PMC: 7616800. DOI: 10.1017/S2045796024000623.

Long-Term Randomized Controlled Trials of Diet Intervention Reports and Their Impact on Cancer: A Systematic Review.

Sauter E, Butera G, Agurs-Collins T Cancers (Basel). 2024; 16(19).

PMID: 39409915 PMC: 11475623. DOI: 10.3390/cancers16193296.

The Aging of Polymers under Electromagnetic Radiation.

Maraveas C, Kyrtopoulos I, Arvanitis K, Bartzanas T Polymers (Basel). 2024; 16(5).

PMID: 38475374 PMC: 10934588. DOI: 10.3390/polym16050689.

Open Science Practices in Psychiatric Genetics: A Primer.

Kepinska A, Johnson J, Huckins L Biol Psychiatry Glob Open Sci. 2024; 4(1):110-119.

PMID: 38298792 PMC: 10829621. DOI: 10.1016/j.bpsgos.2023.08.007.

References

Hughes R, Heron J, Sterne J, Tilling K . Accounting for missing data in statistical analyses: multiple imputation is not always the answer. Int J Epidemiol. 2019; 48(4):1294-1304. PMC: 6693809. DOI: 10.1093/ije/dyz032. View

Steegen S, Tuerlinckx F, Gelman A, Vanpaemel W . Increasing Transparency Through a Multiverse Analysis. Perspect Psychol Sci. 2016; 11(5):702-712. DOI: 10.1177/1745691616658637. View

Botvinik-Nezer R, Holzmeister F, Camerer C, Dreber A, Huber J, Johannesson M . Variability in the analysis of a single neuroimaging dataset by many teams. Nature. 2020; 582(7810):84-88. PMC: 7771346. DOI: 10.1038/s41586-020-2314-9. View

Rubin D . The design versus the analysis of observational studies for causal effects: parallels with the design of randomized trials. Stat Med. 2006; 26(1):20-36. DOI: 10.1002/sim.2739. View

Lash T . Preregistration of study protocols is unlikely to improve the yield from our science, but other strategies might. Epidemiology. 2010; 21(5):612-3. DOI: 10.1097/EDE.0b013e3181e9bba6. View

De Vries Y, Roest A, de Jonge P, Cuijpers P, Munafo M, Bastiaansen J . The cumulative effect of reporting and citation biases on the apparent efficacy of treatments: the case of depression. Psychol Med. 2018; 48(15):2453-2455. PMC: 6190062. DOI: 10.1017/S0033291718001873. View

Nosek B, Beck E, Campbell L, Flake J, Hardwicke T, Mellor D . Preregistration Is Hard, And Worthwhile. Trends Cogn Sci. 2019; 23(10):815-818. DOI: 10.1016/j.tics.2019.07.009. View

Szollosi A, Kellen D, Navarro D, Shiffrin R, van Rooij I, Van Zandt T . Is Preregistration Worthwhile?. Trends Cogn Sci. 2020; 24(2):94-95. DOI: 10.1016/j.tics.2019.11.009. View

Goldstein B . Five analytic challenges in working with electronic health records data to support clinical trials with some solutions. Clin Trials. 2020; 17(4):370-376. DOI: 10.1177/1740774520931211. View

10.

Franco A, Malhotra N, Simonovits G . Social science. Publication bias in the social sciences: unlocking the file drawer. Science. 2014; 345(6203):1502-5. DOI: 10.1126/science.1255484. View

11.

Greenland S . Invited Commentary: The Need for Cognitive Science in Methodology. Am J Epidemiol. 2017; 186(6):639-645. DOI: 10.1093/aje/kwx259. View

12.

Lakens D . Equivalence Tests: A Practical Primer for Tests, Correlations, and Meta-Analyses. Soc Psychol Personal Sci. 2017; 8(4):355-362. PMC: 5502906. DOI: 10.1177/1948550617697177. View

13.

Nuzzo R . How scientists fool themselves - and how they can stop. Nature. 2015; 526(7572):182-5. DOI: 10.1038/526182a. View

14.

Thompson W, Wright J, Bissett P, Poldrack R . Dataset decay and the problem of sequential analyses on open datasets. Elife. 2020; 9. PMC: 7237204. DOI: 10.7554/eLife.53498. View

15.

Lawlor D . Quality in epidemiological research: should we be submitting papers before we have the results and submitting more hypothesis-generating research?. Int J Epidemiol. 2007; 36(5):940-3. DOI: 10.1093/ije/dym168. View

16.

Schonbrodt F, Wagenmakers E . Bayes factor design analysis: Planning for compelling evidence. Psychon Bull Rev. 2017; 25(1):128-142. DOI: 10.3758/s13423-017-1230-y. View

17.

Border R, Johnson E, Evans L, Smolen A, Berley N, Sullivan P . No Support for Historical Candidate Gene or Candidate Gene-by-Interaction Hypotheses for Major Depression Across Multiple Large Samples. Am J Psychiatry. 2019; 176(5):376-387. PMC: 6548317. DOI: 10.1176/appi.ajp.2018.18070881. View

18.

Mathieu S, Boutron I, Moher D, Altman D, Ravaud P . Comparison of registered and published primary outcomes in randomized controlled trials. JAMA. 2009; 302(9):977-84. DOI: 10.1001/jama.2009.1242. View

19.

Pingault J, OReilly P, Schoeler T, Ploubidis G, Rijsdijk F, Dudbridge F . Using genetic data to strengthen causal inference in observational research. Nat Rev Genet. 2018; 19(9):566-580. DOI: 10.1038/s41576-018-0020-3. View

20.

Wagenmakers E, Wetzels R, Borsboom D, van der Maas H, Kievit R . An Agenda for Purely Confirmatory Research. Perspect Psychol Sci. 2015; 7(6):632-8. DOI: 10.1177/1745691612463078. View