Assessment of Racial Bias Within the Risk Analysis Index of Frailty

Overview

Journal Ann Surg Open

Publisher Wolters Kluwer

Specialty General Surgery

Date 2024 Dec 23

PMID 39711679

Authors

Michael A Jacobs

Leslie R M Hausmann

Robert M Handzel

Susanne Schmidt

Carly A Jacobs

Daniel E Hall

Affiliations

Soon will be listed here.

Abstract

Objective: Our objective was to assess potential racial bias within the Risk Analysis Index (RAI).

Background: Patient risk measures are rarely tested for racial bias. Measures of frailty, like the RAI, need to be evaluated for poor predictive performance among Black patients.

Methods: Retrospective cohort study using April 2010-March 2019 Veterans Affairs Surgical Quality Improvement Program and 2010-2019 National Surgical Quality Improvement Program data. The performance of the RAI and several potential variants were compared between Black and White cases using various metrics to predict mortality (180-day for Veterans Affairs Surgical Quality Improvement Program, 30-day for National Surgical Quality Improvement Program).

Results: Using the current, clinical threshold, the RAI performed as good or better among Black cases across various performance metrics White. When a higher threshold was used, Black cases had higher true positive rates but lower true negative rates, yielding 2.0% higher balanced accuracy. No RAI variant noticeably eliminated bias, improved parity across both true positives and true negatives, or improved overall model performance.

Conclusions: The RAI tends to predict mortality among Black patients better than it predicts mortality among White patients. As existing bias-reducing techniques were not effective, further research into bias-reducing techniques is needed, especially for clinical risk predictions. We recommend using the RAI for both statistical analysis of surgical cohorts and quality improvement programs, such as the Surgical Pause.

References

Arya S, Varley P, Youk A, Borrebach J, Perez S, Massarweh N . Recalibration and External Validation of the Risk Analysis Index: A Surgical Frailty Assessment Tool. Ann Surg. 2019; 272(6):996-1005. PMC: 8785437. DOI: 10.1097/SLA.0000000000003276. View

George E, Hall D, Youk A, Chen R, Kashikar A, Trickey A . Association Between Patient Frailty and Postoperative Mortality Across Multiple Noncardiac Surgical Specialties. JAMA Surg. 2020; 156(1):e205152. PMC: 7675216. DOI: 10.1001/jamasurg.2020.5152. View

Perkins N, Schisterman E . The inconsistency of "optimal" cutpoints obtained using two criteria based on the receiver operating characteristic curve. Am J Epidemiol. 2006; 163(7):670-5. PMC: 1444894. DOI: 10.1093/aje/kwj063. View

Chouldechova A . Fair Prediction with Disparate Impact: A Study of Bias in Recidivism Prediction Instruments. Big Data. 2017; 5(2):153-163. DOI: 10.1089/big.2016.0047. View

Charlson M, Pompei P, Ales K, MacKenzie C . A new method of classifying prognostic comorbidity in longitudinal studies: development and validation. J Chronic Dis. 1987; 40(5):373-83. DOI: 10.1016/0021-9681(87)90171-8. View

Pierson E . Accuracy and Equity in Clinical Risk Prediction. N Engl J Med. 2024; 390(2):100-102. DOI: 10.1056/NEJMp2311050. View

Hall D, Arya S, Schmid K, Carlson M, Lavedan P, Bailey T . Association of a Frailty Screening Initiative With Postoperative Survival at 30, 180, and 365 Days. JAMA Surg. 2016; 152(3):233-240. PMC: 7180387. DOI: 10.1001/jamasurg.2016.4219. View

Hsu J, Price M, Vogeli C, Brand R, Chernew M, Chaguturu S . Bending The Spending Curve By Altering Care Delivery Patterns: The Role Of Care Management Within A Pioneer ACO. Health Aff (Millwood). 2017; 36(5):876-884. DOI: 10.1377/hlthaff.2016.0922. View

Massarweh N, Kaji A, Itani K . Practical Guide to Surgical Data Sets: Veterans Affairs Surgical Quality Improvement Program (VASQIP). JAMA Surg. 2018; 153(8):768-769. DOI: 10.1001/jamasurg.2018.0504. View

10.

Geronimus A, Bound J, Waidmann T, Hillemeier M, Burns P . Excess mortality among blacks and whites in the United States. N Engl J Med. 1996; 335(21):1552-8. DOI: 10.1056/NEJM199611213352102. View

11.

Yan Q, Kim J, Hall D, Shinall Jr M, Reitz K, Stitzenberg K . Association of Frailty and the Expanded Operative Stress Score with Preoperative Acute Serious Conditions, Complications, and Mortality in Males Compared to Females: A Retrospective Observational Study. Ann Surg. 2021; 277(2):e294-e304. PMC: 8709872. DOI: 10.1097/SLA.0000000000005027. View

12.

Astone N, Ensminger M, Juon H . Early adult characteristics and mortality among inner-city African American women. Am J Public Health. 2002; 92(4):640-5. PMC: 1447130. DOI: 10.2105/ajph.92.4.640. View

13.

Matthews B . Comparison of the predicted and observed secondary structure of T4 phage lysozyme. Biochim Biophys Acta. 1975; 405(2):442-51. DOI: 10.1016/0005-2795(75)90109-9. View

14.

Dicpinigaitis A, Khamzina Y, Hall D, Nassereldine H, Kennedy J, Seymour C . Adaptation of the Risk Analysis Index for Frailty Assessment Using Diagnostic Codes. JAMA Netw Open. 2024; 7(5):e2413166. PMC: 11127118. DOI: 10.1001/jamanetworkopen.2024.13166. View

15.

Shinall Jr M, Arya S, Youk A, Varley P, Shah R, Massarweh N . Association of Preoperative Patient Frailty and Operative Stress With Postoperative Mortality. JAMA Surg. 2019; 155(1):e194620. PMC: 6865246. DOI: 10.1001/jamasurg.2019.4620. View

16.

Best W, Khuri S, Phelan M, Hur K, Henderson W, Demakis J . Identifying patient preoperative risk factors and postoperative adverse events in administrative databases: results from the Department of Veterans Affairs National Surgical Quality Improvement Program. J Am Coll Surg. 2002; 194(3):257-66. DOI: 10.1016/s1072-7515(01)01183-8. View

17.

Yuan C, Linn K, Hubbard R . Algorithmic Fairness of Machine Learning Models for Alzheimer Disease Progression. JAMA Netw Open. 2023; 6(11):e2342203. PMC: 10630899. DOI: 10.1001/jamanetworkopen.2023.42203. View

18.

Gagne J, Glynn R, Avorn J, Levin R, Schneeweiss S . A combined comorbidity score predicted mortality in elderly patients better than existing scores. J Clin Epidemiol. 2011; 64(7):749-59. PMC: 3100405. DOI: 10.1016/j.jclinepi.2010.10.004. View

19.

Ghaferi A, Schwartz T, Pawlik T . STROBE Reporting Guidelines for Observational Studies. JAMA Surg. 2021; 156(6):577-578. DOI: 10.1001/jamasurg.2021.0528. View

20.

Vaughan Sarrazin M . Balancing Statistical Precision With Societal Goals to Reduce Health Disparities Using Clinical Support Tools. JAMA Netw Open. 2023; 6(9):e2331140. DOI: 10.1001/jamanetworkopen.2023.31140. View