Benchmarking Survival Outcomes: A Funnel Plot for Survival Data

Overview

Journal Stat Methods Med Res

Publisher Sage Publications

Specialties Public Health
Science

Date 2022 Mar 8

PMID 35257603

Authors

Hein Putter

Dirk-Jan Eikema

Liesbeth C de Wreede

Eoin McGrath

Isabel Sanchez-Ortega

Riccardo Saccardi

John A Snowden

Erik W van Zwet

Affiliations

Soon will be listed here.

Abstract

Benchmarking is commonly used in many healthcare settings to monitor clinical performance, with the aim of increasing cost-effectiveness and safe care of patients. The funnel plot is a popular tool in visualizing the performance of a healthcare center in relation to other centers and to a target, taking into account statistical uncertainty. In this paper, we develop a methodology for constructing funnel plots for survival data. The method takes into account censoring and can deal with differences in censoring distributions across centers. Practical issues in implementing the methodology are discussed, particularly in the setting of benchmarking clinical outcomes for hematopoietic stem cell transplantation. A simulation study is performed to assess the performance of the funnel plots under several scenarios. Our methodology is illustrated using data from the European Society for Blood and Marrow Transplantation benchmarking project.

Citing Articles

The association between hospital volume and overall survival in adult AML patients treated with intensive chemotherapy.

Kaplan Z, van Leeuwen N, van Klaveren D, Eijkenaar F, Visser O, Posthuma E ESMO Open. 2025; 10(2):104152.

PMID: 39889323 PMC: 11833631. DOI: 10.1016/j.esmoop.2025.104152.

Predictors and Significance of Readmission after Esophagogastric Surgery: A Nationwide Analysis.

Evans R, Kamarajah S, Evison F, Zou X, Coupland B, Griffiths E Ann Surg Open. 2024; 5(1):e363.

PMID: 38883942 PMC: 11175914. DOI: 10.1097/AS9.0000000000000363.

Benchmarking of survival outcomes following Haematopoietic Stem Cell Transplantation (HSCT): an update of the ongoing project of the European Society for Blood and Marrow Transplantation (EBMT) and Joint Accreditation Committee of ISCT and EBMT....

Saccardi R, Putter H, Eikema D, Busto M, McGrath E, Middelkoop B Bone Marrow Transplant. 2023; 58(6):659-666.

PMID: 36894635 PMC: 9995719. DOI: 10.1038/s41409-023-01924-6.

References

VanderWeele T . Principles of confounder selection. Eur J Epidemiol. 2019; 34(3):211-219. PMC: 6447501. DOI: 10.1007/s10654-019-00494-6. View

Kunadian B, Dunning J, Roberts A, Morley R, Twomey D, Hall J . Cumulative funnel plots for the early detection of interoperator variation: retrospective database analysis of observed versus predicted results of percutaneous coronary intervention. BMJ. 2008; 336(7650):931-4. PMC: 2335227. DOI: 10.1136/bmj.39512.529120.BE. View

Berry G . The analysis of mortality by the subject-years method. Biometrics. 1983; 39(1):173-84. View

Manktelow B, Seaton S, Evans T . Funnel plot control limits to identify poorly performing healthcare providers when there is uncertainty in the value of the benchmark. Stat Methods Med Res. 2014; 25(6):2670-2684. DOI: 10.1177/0962280214530281. View

Jacobs J, OBrien S, Pasquali S, Jacobs M, Lacour-Gayet F, Tchervenkov C . Variation in outcomes for benchmark operations: an analysis of the Society of Thoracic Surgeons Congenital Heart Surgery Database. Ann Thorac Surg. 2011; 92(6):2184-91. PMC: 3263755. DOI: 10.1016/j.athoracsur.2011.06.008. View

van Houwelingen H, Putter H . Comparison of stopped Cox regression with direct methods such as pseudo-values and binomial regression. Lifetime Data Anal. 2014; 21(2):180-96. DOI: 10.1007/s10985-014-9299-3. View

Logan B, Nelson G, Klein J . Analyzing center specific outcomes in hematopoietic cell transplantation. Lifetime Data Anal. 2008; 14(4):389-404. PMC: 2709496. DOI: 10.1007/s10985-008-9100-6. View

Quaresma M, Coleman M, Rachet B . Funnel plots for population-based cancer survival: principles, methods and applications. Stat Med. 2013; 33(6):1070-80. DOI: 10.1002/sim.5953. View

Clark T, Altman D, De Stavola B . Quantification of the completeness of follow-up. Lancet. 2002; 359(9314):1309-10. DOI: 10.1016/s0140-6736(02)08272-7. View

10.

Snowden J, Saccardi R, Orchard K, Ljungman P, Duarte R, Labopin M . Benchmarking of survival outcomes following haematopoietic stem cell transplantation: A review of existing processes and the introduction of an international system from the European Society for Blood and Marrow Transplantation (EBMT) and the Joint.... Bone Marrow Transplant. 2019; 55(4):681-694. PMC: 7113189. DOI: 10.1038/s41409-019-0718-7. View

11.

Mayer E, Bottle A, Rao C, Darzi A, Athanasiou T . Funnel plots and their emerging application in surgery. Ann Surg. 2009; 249(3):376-83. DOI: 10.1097/SLA.0b013e31819a47b1. View

12.

Klein J, Logan B, Harhoff M, Andersen P . Analyzing survival curves at a fixed point in time. Stat Med. 2007; 26(24):4505-19. DOI: 10.1002/sim.2864. View

13.

Verburg I, Holman R, Peek N, Abu-Hanna A, de Keizer N . Guidelines on constructing funnel plots for quality indicators: A case study on mortality in intensive care unit patients. Stat Methods Med Res. 2017; 27(11):3350-3366. PMC: 6193208. DOI: 10.1177/0962280217700169. View

14.

Spiegelhalter D . Funnel plots for comparing institutional performance. Stat Med. 2004; 24(8):1185-202. DOI: 10.1002/sim.1970. View

15.

Armand P, Kim H, Logan B, Wang Z, Alyea E, Kalaycio M . Validation and refinement of the Disease Risk Index for allogeneic stem cell transplantation. Blood. 2014; 123(23):3664-71. PMC: 4047501. DOI: 10.1182/blood-2014-01-552984. View