Identification of Tools Used to Assess the External Validity of Randomized Controlled Trials in Reviews: a Systematic Review of Measurement Properties

Overview

Journal BMC Med Res Methodol

Publisher Biomed Central

Specialties General Medicine
Health Services

Date 2022 Apr 7

PMID 35387582

Authors

Andres Jung

Julia Balzer

Tobias Braun

Kerstin Luedtke

Affiliations

Soon will be listed here.

Abstract

Background: Internal and external validity are the most relevant components when critically appraising randomized controlled trials (RCTs) for systematic reviews. However, there is no gold standard to assess external validity. This might be related to the heterogeneity of the terminology as well as to unclear evidence of the measurement properties of available tools. The aim of this review was to identify tools to assess the external validity of RCTs. It was further, to evaluate the quality of identified tools and to recommend the use of individual tools to assess the external validity of RCTs in future systematic reviews.

Methods: A two-phase systematic literature search was performed in four databases: PubMed, Scopus, PsycINFO via OVID, and CINAHL via EBSCO. First, tools to assess the external validity of RCTs were identified. Second, studies investigating the measurement properties of these tools were selected. The measurement properties of each included tool were appraised using an adapted version of the COnsensus based Standards for the selection of health Measurement INstruments (COSMIN) guidelines.

Results: 38 publications reporting on the development or validation of 28 included tools were included. For 61% (17/28) of the included tools, there was no evidence for measurement properties. For the remaining tools, reliability was the most frequently assessed property. Reliability was judged as "sufficient" for three tools (very low certainty of evidence). Content validity was rated as "sufficient" for one tool (moderate certainty of evidence).

Conclusions: Based on these results, no available tool can be fully recommended to assess the external validity of RCTs in systematic reviews. Several steps are required to overcome the identified difficulties to either adapt and validate available tools or to develop a better suitable tool.

Trial Registration: Prospective registration at Open Science Framework (OSF): https://doi.org/10.17605/OSF.IO/PTG4D .

Citing Articles

Methodological review to develop a list of bias items for adaptive clinical trials: Protocol and rationale.

Staibano P, McKechnie T, Thabane A, Olteanu D, Nanji K, Zhang H PLoS One. 2024; 19(12):e0303315.

PMID: 39666716 PMC: 11637403. DOI: 10.1371/journal.pone.0303315.

Randomized controlled trials comparing gastric bypass, gastric band, and sleeve gastrectomy: A systematic review examining validity and applicability to wider clinical practice.

Chalmers K, Cousins S, Blazeby J Obes Rev. 2024; 25(5):e13718.

PMID: 38346786 PMC: 11478934. DOI: 10.1111/obr.13718.

Towards a framework for systematic reviews of the prevalence of exposure to environmental and occupational risk factors.

Pega F, Momen N, Bero L, Whaley P Environ Health. 2022; 21(1):64.

PMID: 35794579 PMC: 9258093. DOI: 10.1186/s12940-022-00878-4.

The Methodological Quality of Studies Investigating the Acute Effects of Exercise During Hypoxia Over the Past 40 years: A Systematic Review.

Hohenauer E, Freitag L, Herten M, Siallagan J, Pollock E, Taube W Front Physiol. 2022; 13:919359.

PMID: 35784889 PMC: 9243659. DOI: 10.3389/fphys.2022.919359.

References

Clegg A, Bryant J, Nicholson T, McIntyre L, De Broe S, Gerard K . Clinical and cost-effectiveness of donepezil, rivastigmine and galantamine for Alzheimer's disease: a rapid and systematic review. Health Technol Assess. 2001; 5(1):1-137. DOI: 10.3310/hta5010. View

Terwee C, Jansma E, Riphagen I, de Vet H . Development of a methodological PubMed search filter for finding studies on measurement properties of measurement instruments. Qual Life Res. 2009; 18(8):1115-23. PMC: 2744791. DOI: 10.1007/s11136-009-9528-5. View

Cho M, Bero L . Instruments for assessing the quality of drug studies published in the medical literature. JAMA. 1994; 272(2):101-4. View

Sierevelt I, Zwiers R, Schats W, Haverkamp D, Terwee C, Nolte P . Measurement properties of the most commonly used Foot- and Ankle-Specific Questionnaires: the FFI, FAOS and FAAM. A systematic review. Knee Surg Sports Traumatol Arthrosc. 2017; 26(7):2059-2073. DOI: 10.1007/s00167-017-4748-7. View

Munthe-Kaas H, Nokleby H, Nguyen L . Systematic mapping of checklists for assessing transferability. Syst Rev. 2019; 8(1):22. PMC: 6330740. DOI: 10.1186/s13643-018-0893-4. View

Clark R, Locke M, Hill B, Wells C, Bialocerkowski A . Clinimetric properties of lower limb neurological impairment tests for children and young people with a neurological condition: A systematic review. PLoS One. 2017; 12(7):e0180031. PMC: 5495217. DOI: 10.1371/journal.pone.0180031. View

Armijo-Olivo S, Stiles C, Hagen N, Biondo P, Cummings G . Assessment of study quality for systematic reviews: a comparison of the Cochrane Collaboration Risk of Bias Tool and the Effective Public Health Practice Project Quality Assessment Tool: methodological research. J Eval Clin Pract. 2010; 18(1):12-8. DOI: 10.1111/j.1365-2753.2010.01516.x. View

Cho M, Bero L . The quality of drug studies published in symposium proceedings. Ann Intern Med. 1996; 124(5):485-9. DOI: 10.7326/0003-4819-124-5-199603010-00004. View

Brozek J, Canelo-Aybar C, Akl E, Bowen J, Bucher J, Chiu W . GRADE Guidelines 30: the GRADE approach to assessing the certainty of modeled evidence-An overview in the context of health decision-making. J Clin Epidemiol. 2020; 129:138-150. PMC: 8514123. DOI: 10.1016/j.jclinepi.2020.09.018. View

10.

Weise A, Buchter R, Pieper D, Mathes T . Assessing context suitability (generalizability, external validity, applicability or transferability) of findings in evidence syntheses in healthcare-An integrative review of methodological guidance. Res Synth Methods. 2020; 11(6):760-779. DOI: 10.1002/jrsm.1453. View

11.

Rothwell P . External validity of randomised controlled trials: "to whom do the results of this trial apply?". Lancet. 2005; 365(9453):82-93. DOI: 10.1016/S0140-6736(04)17670-8. View

12.

Mokkink L, Terwee C, Patrick D, Alonso J, Stratford P, Knol D . The COSMIN checklist for assessing the methodological quality of studies on measurement properties of health status measurement instruments: an international Delphi study. Qual Life Res. 2010; 19(4):539-49. PMC: 2852520. DOI: 10.1007/s11136-010-9606-8. View

13.

Dekkers O, Bossuyt P, Vandenbroucke J . How trial results are intended to be used: is PRECIS-2 a step forward?. J Clin Epidemiol. 2016; 84:25-26. DOI: 10.1016/j.jclinepi.2016.01.033. View

14.

OConnor S, Tully M, Ryan B, Bradley J, Baxter G, McDonough S . Failure of a numerical quality assessment scale to identify potential risk of bias in a systematic review: a comparison study. BMC Res Notes. 2015; 8:224. PMC: 4467625. DOI: 10.1186/s13104-015-1181-1. View

15.

Mokkink L, Terwee C, Patrick D, Alonso J, Stratford P, Knol D . The COSMIN study reached international consensus on taxonomy, terminology, and definitions of measurement properties for health-related patient-reported outcomes. J Clin Epidemiol. 2010; 63(7):737-45. DOI: 10.1016/j.jclinepi.2010.02.006. View

16.

Meader N, King K, Llewellyn A, Norman G, Brown J, Rodgers M . A checklist designed to aid consistency and reproducibility of GRADE assessments: development and pilot validation. Syst Rev. 2014; 3:82. PMC: 4124503. DOI: 10.1186/2046-4053-3-82. View

17.

Maassen S, Weggelaar Jansen A, Brekelmans G, Vermeulen H, Oostveen C . Psychometric evaluation of instruments measuring the work environment of healthcare professionals in hospitals: a systematic literature review. Int J Qual Health Care. 2020; 32(8):545-557. PMC: 7654380. DOI: 10.1093/intqhc/mzaa072. View

18.

Bialocerkowski A, Oshea K, Pin T . Psychometric properties of outcome measures for children and adolescents with brachial plexus birth palsy: a systematic review. Dev Med Child Neurol. 2013; 55(12):1075-88. DOI: 10.1111/dmcn.12194. View

19.

Hariton E, Locascio J . Randomised controlled trials - the gold standard for effectiveness research: Study design: randomised controlled trials. BJOG. 2018; 125(13):1716. PMC: 6235704. DOI: 10.1111/1471-0528.15199. View

20.

Downs S, Black N . The feasibility of creating a checklist for the assessment of the methodological quality both of randomised and non-randomised studies of health care interventions. J Epidemiol Community Health. 1998; 52(6):377-84. PMC: 1756728. DOI: 10.1136/jech.52.6.377. View