Effects of Reliability Indicators on Usage, Acceptance and Preference of Predictive Process Management Decision Support Systems

Overview

Journal Qual User Exp

Date 2022 Sep 12

PMID 36092253

Authors

Peter Frohlich

Alexander G Mirnig

Damiano Falcioni

Johann Schrammel

Lisa Diamond

Isabel Fischer

Manfred Tscheligi

Affiliations

Soon will be listed here.

Abstract

Despite the growing availability of data, simulation technologies, and predictive analytics, it is not yet clear whether and under which conditions users will trust Decision Support Systems (DSS). DSS are designed to support users in making more informed decisions in specialized tasks through more accurate predictions and recommendations. This mixed-methods user study contributes to the research on trust calibration by analyzing the potential effects of integrated reliability indication in DSS user interfaces for process management in first-time usage situations characterized by uncertainty. Ten experts specialized in digital tools for construction were asked to test and assess two versions of a DSS in a renovation project scenario. We found that while users stated that they need full access to all information to make their own decisions, reliability indication in DSS tends to make users more willing to make preliminary decisions, with users adapting their confidence and reliance to the indicated reliability. Reliability indication in DSS also increases subjective usefulness and system reliability. Based on these findings, it is recommended that for the design of reliability indication practitioners consider displaying a combination of reliability information at several granularity levels in DSS user interfaces, including visualizations, such as a traffic light system, and to also provide explanations for the reliability information. Further research directions towards achieving trustworthy decision support in complex environments are proposed.

References

Lee J, See K . Trust in automation: designing for appropriate reliance. Hum Factors. 2004; 46(1):50-80. DOI: 10.1518/hfes.46.1.50_30392. View

McGuirl J, Sarter N . Supporting trust calibration and the effective use of decision aids by presenting dynamic system confidence information. Hum Factors. 2007; 48(4):656-65. DOI: 10.1518/001872006779166334. View

Fujimori R, Liu K, Soeno S, Naraba H, Ogura K, Hara K . Acceptance, Barriers, and Facilitators to Implementing Artificial Intelligence-Based Decision Support Systems in Emergency Departments: Quantitative and Qualitative Evaluation. JMIR Form Res. 2022; 6(6):e36501. PMC: 9237770. DOI: 10.2196/36501. View

Ameller J, Rinaudo J, Merly C . The Contribution of Economic Science to Brownfield Redevelopment: A Review. Integr Environ Assess Manag. 2019; 16(2):184-196. DOI: 10.1002/ieam.4233. View

Neyedli H, Hollands J, Jamieson G . Beyond identity: incorporating system reliability information into an automated combat identification system. Hum Factors. 2011; 53(4):338-55. DOI: 10.1177/0018720811413767. View

Sacha D, Senaratne H, Kwon B, Ellis G, Keim D . The Role of Uncertainty, Awareness, and Trust in Visual Analytics. IEEE Trans Vis Comput Graph. 2015; 22(1):240-9. DOI: 10.1109/TVCG.2015.2467591. View

McNeil B, Pauker S, Sox Jr H, Tversky A . On the elicitation of preferences for alternative therapies. N Engl J Med. 1982; 306(21):1259-62. DOI: 10.1056/NEJM198205273062103. View

Wang L, Jamieson G, Hollands J . Trust and reliance on an automated combat identification system. Hum Factors. 2009; 51(3):281-91. DOI: 10.1177/0018720809338842. View

Ognjanovic S, Thuring M, Murphy R, Holscher C . Display clutter and its effects on visual attention distribution and financial risk judgment. Appl Ergon. 2019; 80:168-174. DOI: 10.1016/j.apergo.2019.05.008. View

10.

Naiseh M, Al-Thani D, Jiang N, Ali R . Explainable recommendation: when design meets trust calibration. World Wide Web. 2021; 24(5):1857-1884. PMC: 8327305. DOI: 10.1007/s11280-021-00916-0. View

11.

Provost F, Fawcett T . Data Science and its Relationship to Big Data and Data-Driven Decision Making. Big Data. 2016; 1(1):51-9. DOI: 10.1089/big.2013.1508. View

12.

Correll M, Gleicher M . Error Bars Considered Harmful: Exploring Alternate Encodings for Mean and Error. IEEE Trans Vis Comput Graph. 2015; 20(12):2142-51. PMC: 6214189. DOI: 10.1109/TVCG.2014.2346298. View

13.

Hammond E, Coulon F, Hallett S, Thomas R, Hardy D, Kingdon A . A critical review of decision support systems for brownfield redevelopment. Sci Total Environ. 2021; 785:147132. DOI: 10.1016/j.scitotenv.2021.147132. View

14.

Lipkus I, Samsa G, Rimer B . General performance on a numeracy scale among highly educated samples. Med Decis Making. 2001; 21(1):37-44. DOI: 10.1177/0272989X0102100105. View

15.

Kouri A, Yamada J, Lam Shin Cheung J, van de Velde S, Gupta S . Do providers use computerized clinical decision support systems? A systematic review and meta-regression of clinical decision support uptake. Implement Sci. 2022; 17(1):21. PMC: 8908582. DOI: 10.1186/s13012-022-01199-3. View

16.

Liberati E, Ruggiero F, Galuppo L, Gorli M, Gonzalez-Lorenzo M, Maraldi M . What hinders the uptake of computerized decision support systems in hospitals? A qualitative study and framework for implementation. Implement Sci. 2017; 12(1):113. PMC: 5602839. DOI: 10.1186/s13012-017-0644-2. View