Meta-evaluation of Meta-analysis: Ten Appraisal Questions for Biologists

Overview

Journal BMC Biol

Publisher Biomed Central

Specialty Biology

Date 2017 Mar 5

PMID 28257642

Citations 125

Authors

Shinichi Nakagawa

Daniel W A Noble

Alistair M Senior

Malgorzata Lagisz

Affiliations

Soon will be listed here.

Abstract

Meta-analysis is a statistical procedure for analyzing the combined data from different studies, and can be a major source of concise up-to-date information. The overall conclusions of a meta-analysis, however, depend heavily on the quality of the meta-analytic process, and an appropriate evaluation of the quality of meta-analysis (meta-evaluation) can be challenging. We outline ten questions biologists can ask to critically appraise a meta-analysis. These questions could also act as simple and accessible guidelines for the authors of meta-analyses. We focus on meta-analyses using non-human species, which we term 'biological' meta-analysis. Our ten questions are aimed at enabling a biologist to evaluate whether a biological meta-analysis embodies 'mega-enlightenment', a 'mega-mistake', or something in between.

Citing Articles

Mini meta-analysis of anticholinesterase actions of atorvastatin, simvastatin and rosuvastatin, and identification of their protein targets in .

Mohammad F, Al-Shalchi R Toxicol Rep. 2025; 14:101958.

PMID: 40026475 PMC: 11869536. DOI: 10.1016/j.toxrep.2025.101958.

Global engineering effects of soil invertebrates on ecosystem functions.

Wu D, Du E, Eisenhauer N, Mathieu J, Chu C Nature. 2025; .

PMID: 39939777 DOI: 10.1038/s41586-025-08594-y.

Same data, different analysts: variation in effect sizes due to analytical decisions in ecology and evolutionary biology.

Gould E, Fraser H, Parker T, Nakagawa S, Griffith S, Vesk P BMC Biol. 2025; 23(1):35.

PMID: 39915771 PMC: 11804095. DOI: 10.1186/s12915-024-02101-x.

Poor hypotheses and research waste in biology: learning from a theory crisis in psychology.

Nakagawa S, Armitage D, Froese T, Yang Y, Lagisz M BMC Biol. 2025; 23(1):33.

PMID: 39901226 PMC: 11792729. DOI: 10.1186/s12915-025-02134-w.

Ticks as vectors of Trypanosomatidae with medical or veterinary interest: Insights and implications from a comprehensive systematic review and meta-analysis.

Kernif T, Medrouh B, Eddaikra N, Oury B, Holzmuller P, Sereno D Heliyon. 2025; 10(24):e40895.

PMID: 39759380 PMC: 11698932. DOI: 10.1016/j.heliyon.2024.e40895.

References

Higgins J, Thompson S . Quantifying heterogeneity in a meta-analysis. Stat Med. 2002; 21(11):1539-58. DOI: 10.1002/sim.1186. View

Liberati A, Altman D, Tetzlaff J, Mulrow C, Gotzsche P, Ioannidis J . The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: explanation and elaboration. PLoS Med. 2009; 6(7):e1000100. PMC: 2707010. DOI: 10.1371/journal.pmed.1000100. View

Ellison A . Repeatability and transparency in ecological research. Ecology. 2010; 91(9):2536-9. DOI: 10.1890/09-0032.1. View

Glass G . Meta-analysis at middle age: a personal history. Res Synth Methods. 2015; 6(3):221-31. DOI: 10.1002/jrsm.1133. View

Lajeunesse M . Meta-analysis and the comparative phylogenetic method. Am Nat. 2009; 174(3):369-81. DOI: 10.1086/603628. View

Borenstein M, Hedges L, Higgins J, Rothstein H . A basic introduction to fixed-effect and random-effects models for meta-analysis. Res Synth Methods. 2015; 1(2):97-111. DOI: 10.1002/jrsm.12. View

Lau J, Ioannidis J, Terrin N, Schmid C, Olkin I . The case of the misleading funnel plot. BMJ. 2006; 333(7568):597-600. PMC: 1570006. DOI: 10.1136/bmj.333.7568.597. View

Simmons J, Nelson L, Simonsohn U . False-positive psychology: undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychol Sci. 2011; 22(11):1359-66. DOI: 10.1177/0956797611417632. View

Garratt M, Nakagawa S, Simons M . Comparative idiosyncrasies in life extension by reduced mTOR signalling and its distinctiveness from dietary restriction. Aging Cell. 2016; 15(4):737-43. PMC: 4933670. DOI: 10.1111/acel.12489. View

10.

Cheung M . Modeling dependent effect sizes with three-level meta-analyses: a structural equation modeling approach. Psychol Methods. 2013; 19(2):211-29. DOI: 10.1037/a0032968. View

11.

Leimu R, Koricheva J . Cumulative meta-analysis: a new tool for detection of temporal trends and publication bias in ecology. Proc Biol Sci. 2004; 271(1551):1961-6. PMC: 1691819. DOI: 10.1098/rspb.2004.2828. View

12.

Ioannidis J . Meta-research: The art of getting it wrong. Res Synth Methods. 2015; 1(3-4):169-84. DOI: 10.1002/jrsm.19. View

13.

Nakagawa S, Freckleton R . Missing inaction: the dangers of ignoring missing data. Trends Ecol Evol. 2008; 23(11):592-6. DOI: 10.1016/j.tree.2008.06.014. View

14.

Jackson D, Riley R, White I . Multivariate meta-analysis: potential and promise. Stat Med. 2011; 30(20):2481-98. PMC: 3470931. DOI: 10.1002/sim.4172. View

15.

Moher D, Liberati A, Tetzlaff J, Altman D . Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. Ann Intern Med. 2009; 151(4):264-9, W64. DOI: 10.7326/0003-4819-151-4-200908180-00135. View

16.

Morrissey M . Meta-analysis of magnitudes, differences and variation in evolutionary parameters. J Evol Biol. 2016; 29(10):1882-1904. DOI: 10.1111/jeb.12950. View

17.

Ioannidis J . Why most published research findings are false. PLoS Med. 2005; 2(8):e124. PMC: 1182327. DOI: 10.1371/journal.pmed.0020124. View

18.

Ioannidis J . The Mass Production of Redundant, Misleading, and Conflicted Systematic Reviews and Meta-analyses. Milbank Q. 2016; 94(3):485-514. PMC: 5020151. DOI: 10.1111/1468-0009.12210. View

19.

Senior A, Gosby A, Lu J, Simpson S, Raubenheimer D . Meta-analysis of variance: an illustration comparing the effects of two dietary interventions on variability in weight. Evol Med Public Health. 2016; 2016(1):244-55. PMC: 4981479. DOI: 10.1093/emph/eow020. View

20.

Lajeunessei M . On the meta-analysis of response ratios for studies with correlated and multi-group designs. Ecology. 2011; 92(11):2049-55. DOI: 10.1890/11-0423.1. View