Synthesis Methods Other Than Meta-analysis Were Commonly Used but Seldom Specified: Survey of Systematic Reviews

Objectives: To examine the specification and use of summary and statistical synthesis methods, focusing on synthesis methods other than meta-analysis.

Study Design And Setting: We coded the specification and use of summary and synthesis methods in 100 randomly sampled systematic reviews (SRs) of public health and health systems interventions published in 2018 from the Health Evidence and Health Systems Evidence databases.

Results: Sixty of the 100 SRs used other synthesis methods for some (27/100) or all syntheses (33/100). Of these, 54/60 used vote counting: three based on direction of effect, 36 on statistical significance, and 15 were unclear. Eight SRs summarized effect estimates (for example, using medians). Seventeen SRs used the term 'narrative synthesis' (or equivalent) without describing methods; in practice 15 of these used vote counting. 58/100 SRs used meta-analysis. In SRs providing a rationale for not proceeding with meta-analysis, the most common reason was due to diversity in study characteristics (33/39).

Conclusion: Statistical synthesis methods other than meta-analysis are commonly used, but few SRs describe the methods. Improved description of methods is required to allow users to appropriately interpret findings, critique methods used and verify the results. Greater awareness of the serious limitations of vote counting based on statistical significance is required.