Evaluating Study Design Rigor in Preclinical Cardiovascular Research: A Replication Study

Overview

Journal bioRxiv

Date 2023 Jul 10

PMID 37425725

Authors

Isaiah C Jimenez

Gabrielle C Montenegro

Keyana Zahiri

Damini Patel

Adrienne Mueller

Affiliations

Soon will be listed here.

Abstract

Background: Methodological rigor remains a priority in preclinical cardiovascular research to ensure experimental reproducibility and high-quality research. Limited reproducibility diminishes the translation of preclinical discoveries into medical practice. In addition, lack of reproducibility fosters uncertainty in the public's acceptance of reported research results.

Methods: We evaluated the reporting of methodological practices in preclinical cardiovascular research studies published in leading scientific journals by screening articles for the inclusion of the following study design elements (SDEs): considering sex as a biological variable, randomization, blinding, and sample size power estimation. We screened for these SDEs across articles regarding preclinical cardiovascular research studies published between 2011 and 2021. We replicated and extended a study published in 2017 by Ramirez et al. We hypothesized a higher SDE inclusion across preclinical studies over time, that preclinical studies that include human and animal substudies within the same study will exhibit greater SDE inclusion than animal-only preclinical studies, and that a difference exists in SDE usage between large and small animal models.

Results: SDE inclusion was low; with 15.2% of animal-only studies including both sexes as a biological variable, 30.4% including randomization, 32.1% including blinding, and 8.2% including sample size estimation. The incorporation of SDEs did not significantly increase over the ten-year timeframe in the screened articles. Randomization and sample size estimation differed significantly between animal and human substudies (corrected p=1.85e-05 and corrected p=3.81e-07, respectively.).

Conclusions: Evidence of methodological rigor varies depending on the study type and model organisms used. From 2011-2021, SDE reporting within preclinical studies has not increased, suggesting more work is needed to foster the inclusion of rigorous study design elements in cardiovascular research.

References

Gosselin R . Insufficient transparency of statistical reporting in preclinical research: a scoping review. Sci Rep. 2021; 11(1):3335. PMC: 7870941. DOI: 10.1038/s41598-021-83006-5. View

Houser S, Margulies K, Murphy A, Spinale F, Francis G, Prabhu S . Animal models of heart failure: a scientific statement from the American Heart Association. Circ Res. 2012; 111(1):131-50. DOI: 10.1161/RES.0b013e3182582523. View

Bacmeister L, Schwarzl M, Warnke S, Stoffers B, Blankenberg S, Westermann D . Inflammation and fibrosis in murine models of heart failure. Basic Res Cardiol. 2019; 114(3):19. DOI: 10.1007/s00395-019-0722-5. View

Lapchak P, Zhang J, Noble-Haeusslein L . RIGOR guidelines: escalating STAIR and STEPS for effective translational research. Transl Stroke Res. 2013; 4(3):279-85. PMC: 3644408. DOI: 10.1007/s12975-012-0209-2. View

OConnor A, Totton S, Cullen J, Ramezani M, Kalivarapu V, Yuan C . The study design elements employed by researchers in preclinical animal experiments from two research domains and implications for automation of systematic reviews. PLoS One. 2018; 13(6):e0199441. PMC: 6023607. DOI: 10.1371/journal.pone.0199441. View

Provencher S, Archer S, Ramirez F, Hibbert B, Paulin R, Boucherat O . Standards and Methodological Rigor in Pulmonary Arterial Hypertension Preclinical and Translational Research. Circ Res. 2018; 122(7):1021-1032. DOI: 10.1161/CIRCRESAHA.117.312579. View

Ramirez F, Motazedian P, Jung R, Di Santo P, MacDonald Z, Moreland R . Methodological Rigor in Preclinical Cardiovascular Studies: Targets to Enhance Reproducibility and Promote Research Translation. Circ Res. 2017; 120(12):1916-1926. PMC: 5466021. DOI: 10.1161/CIRCRESAHA.117.310628. View

Collins F, Tabak L . Policy: NIH plans to enhance reproducibility. Nature. 2014; 505(7485):612-3. PMC: 4058759. DOI: 10.1038/505612a. View

Williams J, Chu H, Lown M, Daniel J, Meckl R, Patel D . Weaknesses in Experimental Design and Reporting Decrease the Likelihood of Reproducibility and Generalization of Recent Cardiovascular Research. Cureus. 2022; 14(1):e21086. PMC: 8825449. DOI: 10.7759/cureus.21086. View

10.

Mlinaric A, Horvat M, Supak Smolcic V . Dealing with the positive publication bias: Why you should really publish your negative results. Biochem Med (Zagreb). 2017; 27(3):030201. PMC: 5696751. DOI: 10.11613/BM.2017.030201. View

11.

Justice M, Dhillon P . Using the mouse to model human disease: increasing validity and reproducibility. Dis Model Mech. 2016; 9(2):101-3. PMC: 4770152. DOI: 10.1242/dmm.024547. View

12.

Jung R, Stotts C, Makwana D, Motazedian P, Di Santo P, Goh C . Methodological Rigor in Preclinical Cardiovascular Research: Contemporary Performance of AHA Scientific Publications. Circ Res. 2021; 129(9):887-889. DOI: 10.1161/CIRCRESAHA.121.319921. View

13.

Hirst J, Howick J, Aronson J, Roberts N, Perera R, Koshiaris C . The need for randomization in animal trials: an overview of systematic reviews. PLoS One. 2014; 9(6):e98856. PMC: 4048216. DOI: 10.1371/journal.pone.0098856. View

14.

Ramirez F, Jung R, Motazedian P, Perry-Nguyen D, Di Santo P, Macdonald Z . Journal Initiatives to Enhance Preclinical Research: Analyses of . Stroke. 2019; 51(1):291-299. PMC: 6924942. DOI: 10.1161/STROKEAHA.119.026564. View

15.

Seok J, Warren H, Cuenca A, Mindrinos M, Baker H, Xu W . Genomic responses in mouse models poorly mimic human inflammatory diseases. Proc Natl Acad Sci U S A. 2013; 110(9):3507-12. PMC: 3587220. DOI: 10.1073/pnas.1222878110. View