Computationally Reproducing Results from Meta-analyses in Ecology and Evolutionary Biology Using Shared Code and Data

Overview

Journal PLoS One

Specialties General Medicine
Science

Date 2024 Mar 13

PMID 38478503

Authors

Steven Kambouris

David P Wilkinson

Eden T Smith

Fiona Fidler

Affiliations

Soon will be listed here.

Abstract

Many journals in ecology and evolutionary biology encourage or require authors to make their data and code available alongside articles. In this study we investigated how often this data and code could be used together, when both were available, to computationally reproduce results published in articles. We surveyed the data and code sharing practices of 177 meta-analyses published in ecology and evolutionary biology journals published between 2015-17: 60% of articles shared data only, 1% shared code only, and 15% shared both data and code. In each of the articles which had shared both (n = 26), we selected a target result and attempted to reproduce it. Using the shared data and code files, we successfully reproduced the targeted results in 27-73% of the 26 articles, depending on the stringency of the criteria applied for a successful reproduction. The results from this sample of meta-analyses in the 2015-17 literature can provide a benchmark for future meta-research studies gauging the computational reproducibility of published research in ecology and evolutionary biology.

Citing Articles

Editorial for Attention, Perception, & Psychophysics.

Ngiam W, Geng J, Shomstein S Atten Percept Psychophys. 2024; 86(8):2565-2568.

PMID: 39482551 DOI: 10.3758/s13414-024-02973-9.

References

Minocher R, Atmaca S, Bavero C, McElreath R, Beheim B . Estimating the reproducibility of social learning research published between 1955 and 2018. R Soc Open Sci. 2021; 8(9):210450. PMC: 8441137. DOI: 10.1098/rsos.210450. View

Paz-Vinas I, Loot G, Stevens V, Blanchet S . Evolutionary processes driving spatial patterns of intraspecific genetic diversity in river ecosystems. Mol Ecol. 2015; 24(18):4586-604. DOI: 10.1111/mec.13345. View

Hardwicke T, Mathur M, MacDonald K, Nilsonne G, Banks G, Kidwell M . Data availability, reusability, and analytic reproducibility: evaluating the impact of a mandatory open data policy at the journal . R Soc Open Sci. 2018; 5(8):180448. PMC: 6124055. DOI: 10.1098/rsos.180448. View

Ronget V, Gaillard J, Coulson T, Garratt M, Gueyffier F, Lega J . Causes and consequences of variation in offspring body mass: meta-analyses in birds and mammals. Biol Rev Camb Philos Soc. 2017; 93(1):1-27. DOI: 10.1111/brv.12329. View

Senior A, Nakagawa S, Lihoreau M, Simpson S, Raubenheimer D . An Overlooked Consequence of Dietary Mixing: A Varied Diet Reduces Interindividual Variance in Fitness. Am Nat. 2015; 186(5):649-59. DOI: 10.1086/683182. View

Fidler F, Chee Y, Wintle B, Burgman M, McCarthy M, Gordon A . Metaresearch for Evaluating Reproducibility in Ecology and Evolution. Bioscience. 2017; 67(3):282-289. PMC: 5384162. DOI: 10.1093/biosci/biw159. View

Fraser H, Barnett A, Parker T, Fidler F . The role of replication studies in ecology. Ecol Evol. 2020; 10(12):5197-5207. PMC: 7319129. DOI: 10.1002/ece3.6330. View

Cruwell S, Apthorp D, Baker B, Colling L, Elson M, Geiger S . What's in a Badge? A Computational Reproducibility Investigation of the Open Data Badge Policy in One Issue of . Psychol Sci. 2023; 34(4):512-522. DOI: 10.1177/09567976221140828. View

Moore A, McPeek M, Rausher M, Rieseberg L, Whitlock M . The need for archiving data in evolutionary biology. J Evol Biol. 2010; 23(4):659-60. DOI: 10.1111/j.1420-9101.2010.01937.x. View

10.

Roche D, Kruuk L, Lanfear R, Binning S . Public Data Archiving in Ecology and Evolution: How Well Are We Doing?. PLoS Biol. 2015; 13(11):e1002295. PMC: 4640582. DOI: 10.1371/journal.pbio.1002295. View

11.

. PSYCHOLOGY. Estimating the reproducibility of psychological science. Science. 2015; 349(6251):aac4716. DOI: 10.1126/science.aac4716. View

12.

Whitlock M, McPeek M, Rausher M, Rieseberg L, Moore A . Data archiving. Am Nat. 2010; 175(2):145-6. DOI: 10.1086/650340. View

13.

Miller S, Jernigan C, Legan A, Miller C, Tumulty J, Walton A . Animal behavior missing from data archives. Trends Ecol Evol. 2021; 36(11):960-963. DOI: 10.1016/j.tree.2021.07.008. View

14.

Anderson J . Plant fitness in a rapidly changing world. New Phytol. 2015; 210(1):81-7. DOI: 10.1111/nph.13693. View

15.

Perkel J . Challenge to scientists: does your ten-year-old code still run?. Nature. 2020; 584(7822):656-658. DOI: 10.1038/d41586-020-02462-7. View

16.

Kimmel K, Avolio M, Ferraro P . Empirical evidence of widespread exaggeration bias and selective reporting in ecology. Nat Ecol Evol. 2023; 7(9):1525-1536. DOI: 10.1038/s41559-023-02144-3. View

17.

Risely A, Klaassen M, Hoye B . Migratory animals feel the cost of getting sick: A meta-analysis across species. J Anim Ecol. 2017; 87(1):301-314. DOI: 10.1111/1365-2656.12766. View

18.

Wood B, Muller R, Brown A . Push button replication: Is impact evaluation evidence for international development verifiable?. PLoS One. 2018; 13(12):e0209416. PMC: 6303036. DOI: 10.1371/journal.pone.0209416. View

19.

Xu X, Medvigy D, Wright S, Kitajima K, Wu J, Albert L . Variations of leaf longevity in tropical moist forests predicted by a trait-driven carbon optimality model. Ecol Lett. 2017; 20(9):1097-1106. DOI: 10.1111/ele.12804. View

20.

Hardwicke T, Bohn M, MacDonald K, Hembacher E, Nuijten M, Peloquin B . Analytic reproducibility in articles receiving open data badges at the journal : an observational study. R Soc Open Sci. 2021; 8(1):201494. PMC: 7890505. DOI: 10.1098/rsos.201494. View