Increasing System Transparency About Medical AI Recommendations May Not Improve Clinical Experts’ Decision Quality

55 Pages Posted: 18 Nov 2021

See all articles by Jeffrey Clement

Jeffrey Clement

Carlson School of Management; Augsburg University

Yuqing Ren

Carlson School of Management

Shawn Curley

University of Minnesota - Minneapolis

Date Written: November 8, 2021

Abstract

Medical AI systems generate personalized recommendations to improve patient care, but it is unclear how system transparency affects how clinicians incorporate AI recommendations into care decisions. We employ mixed methods, combining semi-structured interviews and two computer-based experiments to examine factors posited to support proper system use. In the interviews from Study 1, clinicians expressed that features like the level of confidence and explanations for AI recommendations would increase adoption, consistent with the general literature on the use of recommender systems. To evaluate this within the clinical context, we conducted a pair of experiments where kidney transplant experts were faced with decision tasks. In Study 2, participants received AI recommendations for drug dosing and were shown (or not) the confidence level and an explanation. In Study 3, participants were shown explanations (or not) and received two patient cases each with either a high- or low-quality AI recommendation. Contrary to theoretical predictions, providing explanations did not uniformly increase adoption of AI recommendations or improve clinical decision quality. Instead, explanations increased adoption of low-quality AI recommendations and decreased the adoption of high-quality recommendations. The results also revealed significant differences in the behaviors of physicians and non-physicians regarding the use of AI advice.

Keywords: artificial intelligence, medical AI, algorithm aversion, explainable AI, healthcare, Clinical Decision Support System, experts, interview, experiment, mixed method  

JEL Classification: M15, I1

Suggested Citation

Clement, Jeffrey and Ren, Yuqing and Curley, Shawn, Increasing System Transparency About Medical AI Recommendations May Not Improve Clinical Experts’ Decision Quality (November 8, 2021). Available at SSRN: https://ssrn.com/abstract=3961156 or http://dx.doi.org/10.2139/ssrn.3961156

Jeffrey Clement (Contact Author)

Carlson School of Management ( email )

19th Avenue South
Minneapolis, MN 55455
United States

Augsburg University ( email )

2211 Riverside Avenue
Minneapolis, MN 55454
United States

Yuqing Ren

Carlson School of Management ( email )

420 Delaware St. SE
Minneapolis, MN 55455
United States
612-625-5242 (Phone)

HOME PAGE: http://www.csom.umn.edu/Page2075.aspx?type=staff&eid=38674251

Shawn Curley

University of Minnesota - Minneapolis ( email )

110 Wulling Hall, 86 Pleasant St, S.E.
308 Harvard Street SE
Minneapolis, MN 55455
United States

Do you have negative results from your research you’d like to share?

Paper statistics

Downloads
186
Abstract Views
1,028
Rank
295,570
PlumX Metrics