Download This Paper
Preprints with The Lancet is a collaboration between The Lancet Group of journals and SSRN to facilitate the open sharing of preprints for early engagement, community comment, and collaboration. Preprints available here are not Lancet publications or necessarily under review with a Lancet journal. These preprints are early-stage research papers that have not been peer-reviewed. The usual SSRN checks and a Lancet-specific check for appropriateness and transparency have been applied. The findings should not be used for clinical or public health decision-making or presented without highlighting these facts. For more information, please see the FAQs.
Clinical Evaluation Framework Using Behavioural & Visual Attention Read-Outs for Explainable AI (XAI)
19 Pages Posted: 5 Jul 2023
More...Abstract
Background: Explainable AI (XAI) is seen as important for clinical AI-driven recommender systems but systematic evaluation of impact is lacking (typically only non-expert populations and proxy tasks). Here, we use a behavioural proxy (eye-tracking) to see how physicians respond to both safe and unsafe AI suggestions and their accompanying explanations. Methods: We performed an observational human-AI interaction study in a simulation suite with 19 intensive care unit (ICU) physicians who assessed patient scenarios under both ‘safe’ and ‘unsafe’ AI conditions while wearing eye-tracking glasses. Prescription decisions were made both pre- and post-reveal of the AI suggestion and four different types of explanation. We used gaze detection as a proxy for where clinician attention was directed during the simulations. Ethics approval (22/HRA/1610). Findings: Unsafe AI suggestions attracted significantly greater attention than safe AI suggestions, though there was no significantly higher attention placed onto any of the four types of explanation during unsafe AI scenarios. Self-reported usefulness of explanations by physicians did not correlate with the level of attention they devoted to the explanations. We also found differences in prescription variability (both with and without XAI) between this high-fidelity simulation format and data from a previous lower-fidelity vignette experiment. Interpretation: We show that it is feasible to perform eye-tracking to evaluate physician interaction with XAI and that, once again, the reliability of self-reports as a marker of how clinicians interact with AI is poor. We found that the response to safe or unsafe AI is identifiably different but the lack of rescue provided by XAI calls into question its utility as a mitigation against clinicians erroneously following poor quality AI advice. This is of critical importance with the rise of generative AI and the potential for hallucinatory (i.e. unsafe) AI recommendations.
Keywords: Artificial Intelligence, Explainable AI, Clinical Decision Support System, Trust in AI, Intensive Care, Sepsis, Medical Device, Human Factors
Suggested Citation: Suggested Citation