Assessment of Methodological Quality of Primary Studies by Systematic Reviews: Results of the Metaquality Cross Sectional Study

Overview

Journal BMJ

Specialty General Medicine

Date 2005 Apr 9

PMID 15817526

Citations 66

Authors

Lorenzo P Moja

Elena Telaro

Roberto DAmico

Ivan Moschetti

Laura Coe

Alessandro Liberati

Affiliations

Soon will be listed here.

Abstract

Objectives: To describe how the methodological quality of primary studies is assessed in systematic reviews and whether the quality assessment is taken into account in the interpretation of results.

Data Sources: Cochrane systematic reviews and systematic reviews in paper based journals.

Study Selection: 965 systematic reviews (809 Cochrane reviews and 156 paper based reviews) published between 1995 and 2002.

Data Synthesis: The methodological quality of primary studies was assessed in 854 of the 965 systematic reviews (88.5%). This occurred more often in Cochrane reviews than in paper based reviews (93.9% v 60.3%, P < 0.0001). Overall, only 496 (51.4%) used the quality assessment in the analysis and interpretation of the results or in their discussion, with no significant differences between Cochrane reviews and paper based reviews (52% v 49%, P = 0.58). The tools and methods used for quality assessment varied widely.

Conclusions: Cochrane reviews fared better than systematic reviews published in paper based journals in terms of assessment of methodological quality of primary studies, although they both largely failed to take it into account in the interpretation of results. Methods for assessment of methodological quality by systematic reviews are still in their infancy and there is substantial room for improvement.

Citing Articles

Intervention effect estimates in randomised controlled trials conducted in primary care versus secondary or tertiary care settings: a meta-epidemiological study.

Dugard A, Tavernier E, Caille A, Dechartres A, Hoang A, Giraudeau B BMC Med Res Methodol. 2022; 22(1):329.

PMID: 36550405 PMC: 9773496. DOI: 10.1186/s12874-022-01815-2.

Treatment completion among justice-involved youth engaged in behavioral health treatment studies in the United States: A systematic review and meta-analysis.

Johnson-Kwochka A, Salgado E, Pederson C, Aalsma M, Salyers M J Clin Transl Sci. 2022; 6(1):e86.

PMID: 36003208 PMC: 9389282. DOI: 10.1017/cts.2022.418.

A Systematic Review of E-Cigarette Marketing Communication: Messages, Communication Channels, and Strategies.

Lyu J, Huang P, Jiang N, Ling P Int J Environ Res Public Health. 2022; 19(15).

PMID: 35954623 PMC: 9367763. DOI: 10.3390/ijerph19159263.

Assessments of risk of bias in systematic reviews of observational nutritional epidemiologic studies are often not appropriate or comprehensive: a methodological study.

Zeraatkar D, Kohut A, Bhasin A, Morassut R, Churchill I, Gupta A BMJ Nutr Prev Health. 2022; 4(2):487-500.

PMID: 35028518 PMC: 8718856. DOI: 10.1136/bmjnph-2021-000248.

The hidden influence of communities in collaborative funding of clinical science.

Vasan K, West J R Soc Open Sci. 2021; 8(8):210072.

PMID: 34457332 PMC: 8385381. DOI: 10.1098/rsos.210072.

References

Verhagen A, de Vet H, de Bie R, Boers M, van den Brandt P . The art of quality assessment of RCTs included in systematic reviews. J Clin Epidemiol. 2001; 54(7):651-4. DOI: 10.1016/s0895-4356(00)00360-7. View

Altman D, Dore C . Randomisation and baseline comparisons in clinical trials. Lancet. 1990; 335(8682):149-53. DOI: 10.1016/0140-6736(90)90014-v. View

Linde K, Scholz M, Ramirez G, Clausius N, Melchart D, Jonas W . Impact of study quality on outcome in placebo-controlled trials of homeopathy. J Clin Epidemiol. 1999; 52(7):631-6. DOI: 10.1016/s0895-4356(99)00048-7. View

Jadad A, Moore R, Carroll D, Jenkinson C, Reynolds D, Gavaghan D . Assessing the quality of reports of randomized clinical trials: is blinding necessary?. Control Clin Trials. 1996; 17(1):1-12. DOI: 10.1016/0197-2456(95)00134-4. View

Lantz C, Nebenzahl E . Behavior and interpretation of the kappa statistic: resolution of the two paradoxes. J Clin Epidemiol. 1996; 49(4):431-4. DOI: 10.1016/0895-4356(95)00571-4. View

Detsky A, Naylor C, ORourke K, McGeer A, Labbe K . Incorporating variations in the quality of individual randomized trials into meta-analysis. J Clin Epidemiol. 1992; 45(3):255-65. DOI: 10.1016/0895-4356(92)90085-2. View

Moher D, Cook D, Eastwood S, Olkin I, Rennie D, Stroup D . Improving the quality of reports of meta-analyses of randomised controlled trials: the QUOROM statement. Quality of Reporting of Meta-analyses. Lancet. 1999; 354(9193):1896-900. DOI: 10.1016/s0140-6736(99)04149-5. View

Juni P, Altman D, Egger M . Systematic reviews in health care: Assessing the quality of controlled clinical trials. BMJ. 2001; 323(7303):42-6. PMC: 1120670. DOI: 10.1136/bmj.323.7303.42. View

Schulz K, Chalmers I, Hayes R, Altman D . Empirical evidence of bias. Dimensions of methodological quality associated with estimates of treatment effects in controlled trials. JAMA. 1995; 273(5):408-12. DOI: 10.1001/jama.273.5.408. View

10.

Cho M, Bero L . Instruments for assessing the quality of drug studies published in the medical literature. JAMA. 1994; 272(2):101-4. View

11.

Colditz G, Miller J, Mosteller F . How study design affects outcomes in comparisons of therapy. I: Medical. Stat Med. 1989; 8(4):441-54. DOI: 10.1002/sim.4780080408. View

12.

Jadad A, Cook D, Jones A, Klassen T, Tugwell P, Moher M . Methodology and reports of systematic reviews and meta-analyses: a comparison of Cochrane reviews with articles published in paper-based journals. JAMA. 1998; 280(3):278-80. DOI: 10.1001/jama.280.3.278. View

13.

Moher D, Jadad A, Tugwell P . Assessing the quality of randomized controlled trials. Current issues and future directions. Int J Technol Assess Health Care. 1996; 12(2):195-208. DOI: 10.1017/s0266462300009570. View

14.

Brown S . Measurement of quality of primary studies for meta-analysis. Nurs Res. 1991; 40(6):352-5. View

15.

McAlister F, Clark H, Van Walraven C, Straus S, Lawson F, Moher D . The medical review article revisited: has the science improved?. Ann Intern Med. 1999; 131(12):947-51. DOI: 10.7326/0003-4819-131-12-199912210-00007. View

16.

Juni P, Egger M . Allocation concealment in clinical trials. JAMA. 2002; 288(19):2407-8; author reply 2408-9. DOI: 10.1001/jama.288.19.2407-jlt1120-4-2. View

17.

Mulrow C . The medical review article: state of the science. Ann Intern Med. 1987; 106(3):485-8. DOI: 10.7326/0003-4819-106-3-485. View

18.

Oxman A, Guyatt G . Guidelines for reading literature reviews. CMAJ. 1988; 138(8):697-703. PMC: 1267776. View

19.

Moher D, Jadad A, Nichol G, Penman M, Tugwell P, Walsh S . Assessing the quality of randomized controlled trials: an annotated bibliography of scales and checklists. Control Clin Trials. 1995; 16(1):62-73. DOI: 10.1016/0197-2456(94)00031-w. View

20.

Counsell C, Clarke M, Slattery J, Sandercock P . The miracle of DICE therapy for acute stroke: fact or fictional product of subgroup analysis?. BMJ. 1994; 309(6970):1677-81. PMC: 2542663. DOI: 10.1136/bmj.309.6970.1677. View