Disagreements in Risk of Bias Assessment for Randomised Controlled Trials Included in More Than One Cochrane Systematic Reviews: a Research on Research Study Using Cross-sectional Design

Overview

Journal BMJ Open

Specialty General Medicine

Date 2019 Apr 4

PMID 30940766

Citations 9

Authors

Lorenzo Bertizzolo

Patrick Bossuyt

Ignacio Atal

Philippe Ravaud

Agnes Dechartres

Affiliations

Soon will be listed here.

Abstract

Objectives: Assess the frequency and reasons for disagreements in risk of bias assessments for randomised controlled trials (RCTs) included in more than one Cochrane review.

Design: Research on research study, using cross-sectional design.

Data Sources: 2796 Cochrane reviews published between March 2011 and September 2014.

Data Selection: RCTs included in more than one review.

Data Extraction: Risk of bias assessment and support for judgement for five key risk of bias items.

Data Synthesis: For each item, we compared risk of bias assessment made in each review and calculated proportion of agreement. Two reviewers independently analysed 50% of all disagreements by comparing support for each judgement with information from study report to evaluate whether disagreements were related to a difference in information (eg, contact the study author) or a difference in interpretation (same support for judgement but different interpretation). They also identified main reasons for different interpretation.

Results: 1604 RCTs were included in more than one review. Proportion of agreement ranged from 57% (770/1348 trials) for incomplete outcome data to 81% for random sequence generation (1193/1466). Most common source of disagreement was difference in interpretation of the same information, ranging from 65% (88/136) for random sequence generation to 90% (56/62) for blinding of participants and personnel. Access to different information explained 32/136 (24%) disagreements for random sequence generation and 38/205 (19%) for allocation concealment. Disagreements related to difference in interpretation were frequently related to incomplete or unclear reporting in the study report (83% of disagreements related to different interpretation for random sequence generation).

Conclusions: Risk of bias judgements of RCTs included in more than one Cochrane review differed substantially. Most disagreements were related to a difference in interpretation of an incomplete or unclear description in the study report. A clearer guidance on common causes of incomplete information may improve agreement.

Citing Articles

Association between HIV and acquisition of rifamycin resistance with first-line TB treatment: a systematic review and meta-analysis.

Zinyakatira N, Ford N, Cox H BMC Infect Dis. 2024; 24(1):657.

PMID: 38956461 PMC: 11218187. DOI: 10.1186/s12879-024-09514-7.

Disagreements in risk of bias assessment for randomized controlled trials in hypertension-related Cochrane reviews.

Yao Y, Shen J, Luo J, Li N, Liao X, Zhang Y Trials. 2024; 25(1):405.

PMID: 38907276 PMC: 11191165. DOI: 10.1186/s13063-024-08145-2.

The effectiveness of personalised surveillance and aftercare in breast cancer follow-up: a systematic review.

van Maaren M, van Hoeve J, Korevaar J, van Hezewijk M, Siemerink E, Zeillemaker A Support Care Cancer. 2024; 32(5):323.

PMID: 38695938 PMC: 11065941. DOI: 10.1007/s00520-024-08530-2.

Photobiomodulation combination therapy as a new insight in neurological disorders: a comprehensive systematic review.

Farazi N, Salehi-Pourmehr H, Farajdokht F, Mahmoudi J, Sadigh-Eteghad S BMC Neurol. 2024; 24(1):101.

PMID: 38504162 PMC: 10949673. DOI: 10.1186/s12883-024-03593-4.

Systematic review of hematophagous arthropods present in cattle in France.

Prudhomme J, Depaquit J, Fite J, Quillery E, Bouhsira E, Lienard E Parasite. 2023; 30:56.

PMID: 38084937 PMC: 10714678. DOI: 10.1051/parasite/2023059.

References

Turner L, Shamseer L, Altman D, Schulz K, Moher D . Does use of the CONSORT Statement impact the completeness of reporting of randomised controlled trials published in medical journals? A Cochrane review. Syst Rev. 2012; 1:60. PMC: 3564748. DOI: 10.1186/2046-4053-1-60. View

Wilkins A . Risk of bias in assessing Risk of Bias. Ophthalmic Physiol Opt. 2016; 37(1):107-109. DOI: 10.1111/opo.12333. View

Dechartres A, Trinquart L, Faber T, Ravaud P . Empirical evaluation of which trial characteristics are associated with treatment effect estimates. J Clin Epidemiol. 2016; 77:24-37. DOI: 10.1016/j.jclinepi.2016.04.005. View

Turner L, Shamseer L, Altman D, Weeks L, Peters J, Kober T . Consolidated standards of reporting trials (CONSORT) and the completeness of reporting of randomised controlled trials (RCTs) published in medical journals. Cochrane Database Syst Rev. 2012; 11:MR000030. PMC: 7386818. DOI: 10.1002/14651858.MR000030.pub2. View

Savovic J, Weeks L, Sterne J, Turner L, Altman D, Moher D . Evaluation of the Cochrane Collaboration's tool for assessing the risk of bias in randomized trials: focus groups, online survey, proposed recommendations and their implementation. Syst Rev. 2014; 3:37. PMC: 4022341. DOI: 10.1186/2046-4053-3-37. View

Jorgensen L, Paludan-Muller A, Laursen D, Savovic J, Boutron I, Sterne J . Evaluation of the Cochrane tool for assessing risk of bias in randomized clinical trials: overview of published comments and analysis of user practice in Cochrane and non-Cochrane reviews. Syst Rev. 2016; 5:80. PMC: 4862216. DOI: 10.1186/s13643-016-0259-8. View

Shamseer L, Hopewell S, Altman D, Moher D, Schulz K . Update on the endorsement of CONSORT by high impact factor journals: a survey of journal "Instructions to Authors" in 2014. Trials. 2016; 17(1):301. PMC: 4921029. DOI: 10.1186/s13063-016-1408-z. View

Hartling L, Bond K, Vandermeer B, Seida J, Dryden D, Rowe B . Applying the risk of bias tool in a systematic review of combination long-acting beta-agonists and inhaled corticosteroids for persistent asthma. PLoS One. 2011; 6(2):e17242. PMC: 3044729. DOI: 10.1371/journal.pone.0017242. View

da Costa B, Beckett B, Diaz A, Resta N, Johnston B, Egger M . Effect of standardized training on the reliability of the Cochrane risk of bias assessment tool: a prospective study. Syst Rev. 2017; 6(1):44. PMC: 5335785. DOI: 10.1186/s13643-017-0441-7. View

10.

Hopewell S, Boutron I, Altman D, Ravaud P . Incorporation of assessments of risk of bias of primary studies in systematic reviews of randomised trials: a cross-sectional study. BMJ Open. 2013; 3(8):e003342. PMC: 3753473. DOI: 10.1136/bmjopen-2013-003342. View

11.

Cook D, Mulrow C, Haynes R . Systematic reviews: synthesis of best evidence for clinical decisions. Ann Intern Med. 1997; 126(5):376-80. DOI: 10.7326/0003-4819-126-5-199703010-00006. View

12.

Armijo-Olivo S, Stiles C, Hagen N, Biondo P, Cummings G . Assessment of study quality for systematic reviews: a comparison of the Cochrane Collaboration Risk of Bias Tool and the Effective Public Health Practice Project Quality Assessment Tool: methodological research. J Eval Clin Pract. 2010; 18(1):12-8. DOI: 10.1111/j.1365-2753.2010.01516.x. View

13.

Altman D, Moher D, Schulz K . Improving the reporting of randomised trials: the CONSORT Statement and beyond. Stat Med. 2012; 31(25):2985-97. DOI: 10.1002/sim.5402. View

14.

Jordan V, Lensen S, Farquhar C . There were large discrepancies in risk of bias tool judgments when a randomized controlled trial appeared in more than one systematic review. J Clin Epidemiol. 2016; 81:72-76. DOI: 10.1016/j.jclinepi.2016.08.012. View

15.

Schulz K, Chalmers I, Hayes R, Altman D . Empirical evidence of bias. Dimensions of methodological quality associated with estimates of treatment effects in controlled trials. JAMA. 1995; 273(5):408-12. DOI: 10.1001/jama.273.5.408. View

16.

Higgins J, Altman D, Gotzsche P, Juni P, Moher D, Oxman A . The Cochrane Collaboration's tool for assessing risk of bias in randomised trials. BMJ. 2011; 343:d5928. PMC: 3196245. DOI: 10.1136/bmj.d5928. View

17.

Propadalo I, Tranfic M, Vuka I, Barcot O, Poklepovic Pericic T, Puljak L . In Cochrane reviews, risk of bias assessments for allocation concealment were frequently not in line with Cochrane's Handbook guidance. J Clin Epidemiol. 2018; 106:10-17. DOI: 10.1016/j.jclinepi.2018.10.002. View

18.

Page M, Higgins J, Clayton G, Sterne J, Hrobjartsson A, Savovic J . Empirical Evidence of Study Design Biases in Randomized Trials: Systematic Review of Meta-Epidemiological Studies. PLoS One. 2016; 11(7):e0159267. PMC: 4939945. DOI: 10.1371/journal.pone.0159267. View

19.

Balk E, Bonis P, Moskowitz H, Schmid C, Ioannidis J, Wang C . Correlation of quality measures with estimates of treatment effect in meta-analyses of randomized controlled trials. JAMA. 2002; 287(22):2973-82. DOI: 10.1001/jama.287.22.2973. View

20.

Armijo-Olivo S, Ospina M, da Costa B, Egger M, Saltaji H, Fuentes J . Poor reliability between Cochrane reviewers and blinded external reviewers when applying the Cochrane risk of bias tool in physical therapy trials. PLoS One. 2014; 9(5):e96920. PMC: 4019638. DOI: 10.1371/journal.pone.0096920. View