Higher Number of Items Associated with Significantly Lower Response Rates in COS Delphi Surveys

Overview

Journal J Clin Epidemiol

Publisher Elsevier

Specialty Public Health

Date 2018 Dec 18

PMID 30557677

Citations 76

Authors

Elizabeth Gargon

Richard Crew

Girvan Burnside

Paula R Williamson

Affiliations

Soon will be listed here.

Abstract

Objectives: The Delphi method is commonly used to achieve consensus in core outcome set (COS) development. It is important to try to maximize response rates to Delphi studies and minimize attrition rates and potential for bias. The factors that impact response rates in a Delphi study used for COS development are unknown. The objective of this study was to explore the impact of design characteristics on response rates in Delphi surveys within COS development.

Methods: Published and ongoing studies that included Delphi to develop a COS were eligible. Second round voting response rates were analyzed, and multilevel linear regression was conducted to investigate whether design characteristics were associated with the response rate.

Results: Thirty-one studies were included. Two characteristics were significantly associated with a lower response rate: larger panels and studies with more items included.

Conclusion: COS developers should pay attention to methods when designing a COS development study; in particular, the size of the panels and the size of the list of outcomes. We identified other potential design characteristics that might influence response rates but were unable to explore them in this analysis. These should be reported in future reports to allow for further investigation.

Citing Articles

The GenderCOS project: study protocol for the development of two international Core Outcome Sets for genital gender affirming surgery.

Roijer P, Vallinga M, Pidgeon T, Ceulemans A, Bakker A, Carriere B Int J Transgend Health. 2025; 26(1):78-87.

PMID: 39981273 PMC: 11837927. DOI: 10.1080/26895269.2023.2288881.

Pyoderma gangrenosum refined elements for core item selection and evaluation (PRECISE-PG): a study protocol for reaching consensus on core outcome domain items in clinical trials of pyoderma gangrenosum.

Tobey T, Ortega-Loayza A, Choe S, Haddadin O, Bardazzi F, Croitoru D Arch Dermatol Res. 2025; 317(1):449.

PMID: 39979653 DOI: 10.1007/s00403-025-03920-w.

Defining optimal orthogeriatric hip fracture care: a delphi consensus approach.

van Bremen H, Seppala L, Gans E, Hegeman J, van der Velde N, Willems H Eur Geriatr Med. 2025; .

PMID: 39909977 DOI: 10.1007/s41999-025-01156-5.

Engaging health system, service and consumer representatives in the co-design of a multi-criteria decision-making framework for commissioning overweight and obesity programs and services.

Thomas L, Robinson S, Burns S, Mitchell H, Begley A Health Res Policy Syst. 2025; 23(1):10.

PMID: 39833865 PMC: 11744878. DOI: 10.1186/s12961-024-01263-y.

How Delphi studies in the health sciences find consensus: a scoping review.

Schifano J, Niederberger M Syst Rev. 2025; 14(1):14.

PMID: 39810238 PMC: 11734368. DOI: 10.1186/s13643-024-02738-3.

References

Hasson F, Keeney S, McKenna H . Research guidelines for the Delphi survey technique. J Adv Nurs. 2000; 32(4):1008-15. View

Leece P, Bhandari M, Sprague S, Swiontkowski M, Schemitsch E, Tornetta P . Internet versus mailed questionnaires: a controlled comparison (2). J Med Internet Res. 2005; 6(4):e39. PMC: 1550620. DOI: 10.2196/jmir.6.4.e39. View

Dwan K, Altman D, Arnaiz J, Bloom J, Chan A, Cronin E . Systematic review of the empirical evidence of study publication bias and outcome reporting bias. PLoS One. 2008; 3(8):e3081. PMC: 2518111. DOI: 10.1371/journal.pone.0003081. View

Chalmers I, Glasziou P . Avoidable waste in the production and reporting of research evidence. Lancet. 2009; 374(9683):86-9. DOI: 10.1016/S0140-6736(09)60329-9. View

Edwards P, Roberts I, Clarke M, DiGuiseppi C, Wentz R, Kwan I . Methods to increase response to postal and electronic questionnaires. Cochrane Database Syst Rev. 2009; (3):MR000008. PMC: 8941848. DOI: 10.1002/14651858.MR000008.pub4. View

Sinha I, Smyth R, Williamson P . Using the Delphi technique to determine which outcomes to measure in clinical trials: recommendations for the future based on a systematic review of existing studies. PLoS Med. 2011; 8(1):e1000393. PMC: 3026691. DOI: 10.1371/journal.pmed.1000393. View

Williamson P, Altman D, Blazeby J, Clarke M, Devane D, Gargon E . Developing core outcome sets for clinical trials: issues to consider. Trials. 2012; 13:132. PMC: 3472231. DOI: 10.1186/1745-6215-13-132. View

Dwan K, Gamble C, Williamson P, Kirkham J . Systematic review of the empirical evidence of study publication bias and outcome reporting bias - an updated review. PLoS One. 2013; 8(7):e66844. PMC: 3702538. DOI: 10.1371/journal.pone.0066844. View

Chan A, Song F, Vickers A, Jefferson T, Dickersin K, Gotzsche P . Increasing value and reducing waste: addressing inaccessible research. Lancet. 2014; 383(9913):257-66. PMC: 4533904. DOI: 10.1016/S0140-6736(13)62296-5. View

10.

Gargon E, Gurung B, Medley N, Altman D, Blazeby J, Clarke M . Choosing important health outcomes for comparative effectiveness research: a systematic review. PLoS One. 2014; 9(6):e99111. PMC: 4059640. DOI: 10.1371/journal.pone.0099111. View

11.

Hirsch M, Duffy J, Kusznir J, Davis C, Plana M, Khan K . Variation in outcome reporting in endometriosis trials: a systematic review. Am J Obstet Gynecol. 2016; 214(4):452-464. DOI: 10.1016/j.ajog.2015.12.039. View

12.

Gorst S, Gargon E, Clarke M, Blazeby J, Altman D, Williamson P . Choosing Important Health Outcomes for Comparative Effectiveness Research: An Updated Review and User Survey. PLoS One. 2016; 11(1):e0146444. PMC: 4718543. DOI: 10.1371/journal.pone.0146444. View

13.

Gorst S, Gargon E, Clarke M, Smith V, Williamson P . Choosing Important Health Outcomes for Comparative Effectiveness Research: An Updated Review and Identification of Gaps. PLoS One. 2016; 11(12):e0168403. PMC: 5156438. DOI: 10.1371/journal.pone.0168403. View

14.

Walters S, Dos Anjos Henriques-Cadby I, Bortolami O, Flight L, Hind D, Jacques R . Recruitment and retention of participants in randomised controlled trials: a review of trials funded and published by the United Kingdom Health Technology Assessment Programme. BMJ Open. 2017; 7(3):e015276. PMC: 5372123. DOI: 10.1136/bmjopen-2016-015276. View

15.

Gargon E, Williamson P, Young B . Improving core outcome set development: qualitative interviews with developers provided pointers to inform guidance. J Clin Epidemiol. 2017; 86:140-152. PMC: 5513443. DOI: 10.1016/j.jclinepi.2017.04.024. View

16.

Williamson P, Altman D, Bagley H, Barnes K, Blazeby J, Brookes S . The COMET Handbook: version 1.0. Trials. 2017; 18(Suppl 3):280. PMC: 5499094. DOI: 10.1186/s13063-017-1978-4. View

17.

Kirkham J, Davis K, Altman D, Blazeby J, Clarke M, Tunis S . Core Outcome Set-STAndards for Development: The COS-STAD recommendations. PLoS Med. 2017; 14(11):e1002447. PMC: 5689835. DOI: 10.1371/journal.pmed.1002447. View

18.

Brookes S, Chalmers K, Avery K, Coulman K, Blazeby J . Impact of question order on prioritisation of outcomes in the development of a core outcome set: a randomised controlled trial. Trials. 2018; 19(1):66. PMC: 5784591. DOI: 10.1186/s13063-017-2405-6. View

19.

Davis K, Gorst S, Harman N, Smith V, Gargon E, Altman D . Choosing important health outcomes for comparative effectiveness research: An updated systematic review and involvement of low and middle income countries. PLoS One. 2018; 13(2):e0190695. PMC: 5810981. DOI: 10.1371/journal.pone.0190695. View

20.

Treweek S, Bevan S, Bower P, Campbell M, Christie J, Clarke M . Trial Forge Guidance 1: what is a Study Within A Trial (SWAT)?. Trials. 2018; 19(1):139. PMC: 5824570. DOI: 10.1186/s13063-018-2535-5. View