How Robust Are Studies in the American Board of Emergency Medicine Maintenance of Certification Lifelong Learning and Self-assessment? An Examination of Fragility and Bias of Included Randomized Controlled Trials

Overview

Journal AEM Educ Train

Publisher Wiley

Specialty Emergency Medicine

Date 2018 Jul 28

PMID 30051045

Citations 1

Authors

Philip J Davis

Michael Butler

Kirk Magee

Brent Thoma

Christopher P Nickson

N Seth Trueger

Affiliations

Soon will be listed here.

Abstract

Objective: Critics have raised concerns regarding the validity of maintenance of certification (MOC) programs. We sought to examine the quality of the randomized controlled trials (RCTs) selected for the lifelong learning and self-assessment (LLS) component of the American Board of Emergency Medicine (ABEM) MOC program.

Methods: We systematically reviewed the ABEM LLS reading lists from 2004 to 2017 to identify RCTs with dichotomous outcomes and superiority designs. A fragility index (FI) was calculated using Fisher's exact test for all statistically significant dichotomous outcomes. Bivariate correlation was performed to examine associations between the FI and RCT study characteristics. Each included study was evaluated with the Cochrane Collaboration risk-of-bias (ROB) tool.

Results: Thirteen superiority RCTs with dichotomous outcomes were included in the 2004-2017 LLS reading lists. Ten had a statistically significant outcome, and the majority were robust and at low ROB. The median trial size was 511 patients (interquartile range [IQR] = 251-1,517), and the median FI was 10 (IQR = 7-18); i.e., if 10 patients in the treatment arm had not had events, the results would not have been statistically significant.

Conclusions: The majority of RCTs included in the LLS are robust and at low ROB.

Citing Articles

How Robust Are Studies in the American Board of Emergency Medicine Maintenance of Certification Lifelong Learning and Self-assessment? An Examination of Fragility and Bias of Included Randomized Controlled Trials.

Davis P, Butler M, Magee K, Thoma B, Nickson C, Trueger N AEM Educ Train. 2018; 1(4):280-286.

PMID: 30051045 PMC: 6001615. DOI: 10.1002/aet2.10045.

References

Seglen P . Citation rates and journal impact factors are not suitable for evaluation of research. Acta Orthop Scand. 1998; 69(3):224-9. DOI: 10.3109/17453679809000920. View

de Gans J, van de Beek D . Dexamethasone in adults with bacterial meningitis. N Engl J Med. 2002; 347(20):1549-56. DOI: 10.1056/NEJMoa021334. View

Roberts I, Yates D, Sandercock P, Farrell B, Wasserberg J, Lomas G . Effect of intravenous corticosteroids on death within 14 days in 10008 adults with clinically significant head injury (MRC CRASH trial): randomised placebo-controlled trial. Lancet. 2004; 364(9442):1321-8. DOI: 10.1016/S0140-6736(04)17188-2. View

Andersen H, Nielsen T, Rasmussen K, Thuesen L, Kelbaek H, Thayssen P . A comparison of coronary angioplasty with fibrinolytic therapy in acute myocardial infarction. N Engl J Med. 2003; 349(8):733-42. DOI: 10.1056/NEJMoa025142. View

Spiro D, Tay K, Arnold D, Dziura J, Baker M, Shapiro E . Wait-and-see prescription for the treatment of acute otitis media: a randomized controlled trial. JAMA. 2006; 296(10):1235-41. DOI: 10.1001/jama.296.10.1235. View

Counselman F, Carius M, Kowalenko T, Battaglioli N, Hobgood C, Jagoda A . The American Board of Emergency Medicine Maintenance of Certification Summit. J Emerg Med. 2015; 49(5):722-8. DOI: 10.1016/j.jemermed.2015.06.058. View

De Backer D, Biston P, Devriendt J, Madl C, Chochrad D, Aldecoa C . Comparison of dopamine and norepinephrine in the treatment of shock. N Engl J Med. 2010; 362(9):779-89. DOI: 10.1056/NEJMoa0907118. View

Yealy D, Kellum J, Huang D, Barnato A, Weissfeld L, Pike F . A randomized trial of protocol-based care for early septic shock. N Engl J Med. 2014; 370(18):1683-93. PMC: 4101700. DOI: 10.1056/NEJMoa1401602. View

Evaniew N, Files C, Smith C, Bhandari M, Ghert M, Walsh M . The fragility of statistically significant findings from randomized trials in spine surgery: a systematic survey. Spine J. 2015; 15(10):2188-97. DOI: 10.1016/j.spinee.2015.06.004. View

10.

Bjornson C, Klassen T, Williamson J, Brant R, Mitton C, Plint A . A randomized trial of a single dose of oral dexamethasone for mild croup. N Engl J Med. 2004; 351(13):1306-13. DOI: 10.1056/NEJMoa033534. View

11.

Freedman S, Adler M, Seshadri R, Powell E . Oral ondansetron for gastroenteritis in a pediatric emergency department. N Engl J Med. 2006; 354(16):1698-705. DOI: 10.1056/NEJMoa055119. View

12.

Walsh M, Srinathan S, McAuley D, Mrkobrada M, Levine O, Ribic C . The statistical significance of randomized controlled trial results is frequently fragile: a case for a Fragility Index. J Clin Epidemiol. 2014; 67(6):622-8. DOI: 10.1016/j.jclinepi.2013.10.019. View

13.

Imazio M, Brucato A, Cemin R, Ferrua S, Maggiolini S, Beqaraj F . A randomized trial of colchicine for acute pericarditis. N Engl J Med. 2013; 369(16):1522-8. DOI: 10.1056/NEJMoa1208536. View

14.

Davis P, Butler M, Magee K, Thoma B, Nickson C, Trueger N . How Robust Are Studies in the American Board of Emergency Medicine Maintenance of Certification Lifelong Learning and Self-assessment? An Examination of Fragility and Bias of Included Randomized Controlled Trials. AEM Educ Train. 2018; 1(4):280-286. PMC: 6001615. DOI: 10.1002/aet2.10045. View

15.

Khan M, Evaniew N, Gichuru M, Habib A, Ayeni O, Bedi A . The Fragility of Statistically Significant Findings From Randomized Trials in Sports Surgery: A Systematic Survey. Am J Sports Med. 2016; 45(9):2164-2170. DOI: 10.1177/0363546516674469. View

16.

Costantino T, Parikh A, Satz W, Fojtik J . Ultrasonography-guided peripheral intravenous access versus traditional approaches in patients with difficult intravenous access. Ann Emerg Med. 2005; 46(5):456-61. DOI: 10.1016/j.annemergmed.2004.12.026. View

17.

Sandhu A, Dudley R, Kazi D . A Cost Analysis of the American Board of Internal Medicine's Maintenance-of-Certification Program. Ann Intern Med. 2015; 163(6):401-8. DOI: 10.7326/M15-1011. View

18.

Sterne J, Davey Smith G . Sifting the evidence-what's wrong with significance tests?. BMJ. 2001; 322(7280):226-31. PMC: 1119478. DOI: 10.1136/bmj.322.7280.226. View

19.

Ridgeon E, Young P, Bellomo R, Mucchetti M, Lembo R, Landoni G . The Fragility Index in Multicenter Randomized Controlled Critical Care Trials. Crit Care Med. 2016; 44(7):1278-84. DOI: 10.1097/CCM.0000000000001670. View

20.

Trueger N, Thoma B, Hsu C, Sullivan D, Peters L, Lin M . The Altmetric Score: A New Measure for Article-Level Dissemination and Impact. Ann Emerg Med. 2015; 66(5):549-53. DOI: 10.1016/j.annemergmed.2015.04.022. View