Efficient Clinical Evaluation of Guideline Quality: Development and Testing of a New Tool

Overview

Journal BMC Med Res Methodol

Publisher Biomed Central

Specialties General Medicine
Health Services

Date 2014 Jun 3

PMID 24885893

Citations 30

Authors

Karen Grimmer

Janine Margarita Dizon

Steve Milanese

Ellena King

Kate Beaton

Olivia Thorpe

Lucylynn Lizarondo

Julie Luker

Zuzana Machotka

Saravana Kumar

Affiliations

Soon will be listed here.

Abstract

Background: Evaluating the methodological quality of clinical practice guidelines is essential before deciding which ones which could best inform policy or practice. One current method of evaluating clinical guideline quality is the research-focused AGREE II instrument. This uses 23 questions scored 1-7, arranged in six domains, which requires at least two independent testers, and uses a formulaic weighted domain scoring system. Following feedback from time-poor clinicians, policy-makers and managers that this instrument did not suit clinical need, we developed and tested a simpler, shorter, binary scored instrument (the iCAHE Guideline Quality Checklist) designed for single users.

Methods: Content and construct validity, inter-tester reliability and clinical utility were tested by comparing the new iCAHE Guideline Quality Checklist with the AGREE II instrument. Firstly the questions and domains in both instruments were compared. Six randomly-selected guidelines on a similar theme were then assessed by three independent testers with different experience in guideline quality assessment, using both instruments. Per guideline, weighted domain and total AGREE II scores were calculated, using the scoring rubric for three testers. Total iCAHE scores were calculated per guideline, per tester. The linear relationship between iCAHE and AGREE II scores was assessed using Pearson r correlation coefficients. Score differences between testers were assessed for the iCAHE Guideline Quality Checklist.

Results: There were congruent questions in each instrument in four domains (Scope & Purpose, Stakeholder involvement, Underlying evidence/Rigour, Clarity). The iCAHE and AGREE II scores were moderate to strongly correlated for the six guidelines. There was generally good agreement between testers for iCAHE scores, irrespective of their experience. The iCAHE instrument was preferred by all testers, and took significantly less time to administer than the AGREE II instrument. However, the use of only three testers and six guidelines compromised study power, rendering this research as pilot investigations of the psychometric properties of the iCAHE instrument.

Conclusion: The iCAHE Guideline Quality Checklist has promising psychometric properties and clinical utility.

Citing Articles

Fetal alcohol spectrum disorder resources for health professionals: a scoping review.

Stubbs T, Cannon L, Carter E, Naanai H, Okurame J, Martiniuk A BMJ Open. 2024; 14(7):e086999.

PMID: 39002966 PMC: 11253770. DOI: 10.1136/bmjopen-2024-086999.

Clinical Practice Guidelines of the Latin American Federation of Endocrinology for the use of vitamin D in the maintenance of bone health: recommendations for the Latin American context.

Gomez O, Campusano C, Cerdas-P S, Mendoza B, Paez-Talero A, de la Pena-Rodriguez M Arch Osteoporos. 2024; 19(1):46.

PMID: 38850469 PMC: 11162390. DOI: 10.1007/s11657-024-01398-z.

Development, Evaluation, and impLemenTation for guideline adaptation: a quality improvement protocol for the DELTA study in global health practice.

Wang S, Zhang Y, Wen Z, Yang Y, Zhang Y, Geng Y Health Res Policy Syst. 2023; 21(1):114.

PMID: 37915056 PMC: 10619317. DOI: 10.1186/s12961-023-01060-z.

Modern Paediatric Emergency Department: Potential Improvements in Light of New Evidence.

Kula R, Popela S, Klucka J, Charwatova D, Djakow J, Stourac P Children (Basel). 2023; 10(4).

PMID: 37189990 PMC: 10137157. DOI: 10.3390/children10040741.

Involvement of methodological experts and the quality of clinical practice guidelines: a critical appraisal of clinical practice guidelines and a questionnaire survey of the development groups in Japan.

Hatakeyama Y, Seto K, Onishi R, Hirata K, Matsumoto K, Wu Y BMJ Open. 2023; 13(5):e063639.

PMID: 37188477 PMC: 10186473. DOI: 10.1136/bmjopen-2022-063639.

References

Grol R, Wensing M . What drives change? Barriers to and incentives for achieving evidence-based practice. Med J Aust. 2004; 180(S6):S57-60. DOI: 10.5694/j.1326-5377.2004.tb05948.x. View

MacDermid J, Brooks D, Solway S, Switzer-McIntyre S, Brosseau L, Graham I . Reliability and validity of the AGREE instrument used by physical therapists in assessment of clinical practice guidelines. BMC Health Serv Res. 2005; 5(1):18. PMC: 555572. DOI: 10.1186/1472-6963-5-18. View

Cabana M, Rand C, Powe N, Wu A, Wilson M, Abboud P . Why don't physicians follow clinical practice guidelines? A framework for improvement. JAMA. 1999; 282(15):1458-65. DOI: 10.1001/jama.282.15.1458. View

Qaseem A, Forland F, Macbeth F, Ollenschlager G, Phillips S, van der Wees P . Guidelines International Network: toward international standards for clinical practice guidelines. Ann Intern Med. 2012; 156(7):525-31. DOI: 10.7326/0003-4819-156-7-201204030-00009. View

Michie S, Johnston M, Abraham C, Lawton R, Parker D, Walker A . Making psychological theory useful for implementing evidence based practice: a consensus approach. Qual Saf Health Care. 2005; 14(1):26-33. PMC: 1743963. DOI: 10.1136/qshc.2004.011155. View

McColl A, Smith H, White P, Field J . General practitioner's perceptions of the route to evidence based medicine: a questionnaire survey. BMJ. 1998; 316(7128):361-5. PMC: 2665572. DOI: 10.1136/bmj.316.7128.361. View

Grimmer-Somers K, Lekkas P, Nyland L, Young A, Kumar S . Perspectives on research evidence and clinical practice: a survey of Australian physiotherapists. Physiother Res Int. 2007; 12(3):147-61. DOI: 10.1002/pri.363. View

Parahoo K . Barriers to, and facilitators of, research utilization among nurses in Northern Ireland. J Adv Nurs. 2000; 31(1):89-98. DOI: 10.1046/j.1365-2648.2000.01256.x. View

Faul F, Erdfelder E, Buchner A, Lang A . Statistical power analyses using G*Power 3.1: tests for correlation and regression analyses. Behav Res Methods. 2009; 41(4):1149-60. DOI: 10.3758/BRM.41.4.1149. View

10.

Barbosa R, Jawa R, Watters J, Knight J, Kerwin A, Winston E . Evaluation and management of mild traumatic brain injury: an Eastern Association for the Surgery of Trauma practice management guideline. J Trauma Acute Care Surg. 2012; 73(5 Suppl 4):S307-14. DOI: 10.1097/TA.0b013e3182701885. View

11.

Kochanek P, Carney N, Adelson P, Ashwal S, Bell M, Bratton S . Guidelines for the acute medical management of severe traumatic brain injury in infants, children, and adolescents--second edition. Pediatr Crit Care Med. 2012; 13 Suppl 1:S1-82. DOI: 10.1097/PCC.0b013e31823f435c. View

12.

Brouwers M, Kho M, Browman G, Burgers J, Cluzeau F, Feder G . AGREE II: advancing guideline development, reporting and evaluation in health care. CMAJ. 2010; 182(18):E839-42. PMC: 3001530. DOI: 10.1503/cmaj.090449. View

13.

Wimpenny P, van Zelm R . Appraising and comparing pressure ulcer guidelines. Worldviews Evid Based Nurs. 2007; 4(1):40-50. DOI: 10.1111/j.1741-6787.2007.00077.x. View

14.

Maher C, Sherrington C, Herbert R, Moseley A, Elkins M . Reliability of the PEDro scale for rating quality of randomized controlled trials. Phys Ther. 2003; 83(8):713-21. View

15.

. Development and validation of an international appraisal instrument for assessing the quality of clinical practice guidelines: the AGREE project. Qual Saf Health Care. 2003; 12(1):18-23. PMC: 1743672. DOI: 10.1136/qhc.12.1.18. View