Expl(Ai)Ned: The Impact of Explainable Artificial Intelligence on Cognitive Processes

45 Pages Posted: 25 Jun 2021 Last revised: 26 Jun 2021

See all articles by Kevin Bauer

Kevin Bauer

Leibniz Institute for Financial Research SAFE

Moritz von Zahn

Goethe University Frankfurt

Oliver Hinz

Goethe University Frankfurt - Faculty of Economics and Business Administration

Date Written: June 16, 2021

Abstract

This paper explores the interplay of feature-based explainable AI (XAI) tech- niques, information processing, and human beliefs. Using a novel experimental protocol, we study the impact of providing users with explanations about how an AI system weighs inputted information to produce individual predictions (LIME) on users’ weighting of information and beliefs about the task-relevance of information. On the one hand, we find that feature-based explanations cause users to alter their mental weighting of available information according to observed explanations. On the other hand, explanations lead to asymmetric belief adjustments that we inter- pret as a manifestation of the confirmation bias. Trust in the prediction accuracy plays an important moderating role for XAI-enabled belief adjustments. Our results show that feature-based XAI does not only superficially influence decisions but re- ally change internal cognitive processes, bearing the potential to manipulate human beliefs and reinforce stereotypes. Hence, the current regulatory efforts that aim at enhancing algorithmic transparency may benefit from going hand in hand with measures ensuring the exclusion of sensitive personal information in XAI systems. Overall, our findings put assertions that XAI is the silver bullet solving all of AI systems’ (black box) problems into perspective.

Keywords: XAI, Explainable machine learning, Information processing, Belief up-dating, Algorithmic transparency

Suggested Citation

Bauer, Kevin and von Zahn, Moritz and Hinz, Oliver, Expl(Ai)Ned: The Impact of Explainable Artificial Intelligence on Cognitive Processes (June 16, 2021). SAFE Working Paper No. 315, Available at SSRN: https://ssrn.com/abstract=3872711 or http://dx.doi.org/10.2139/ssrn.3872711

Kevin Bauer (Contact Author)

Leibniz Institute for Financial Research SAFE ( email )

(http://www.safe-frankfurt.de)
Theodor-W.-Adorno-Platz 3
Frankfurt am Main, 60323
Germany

Moritz Von Zahn

Goethe University Frankfurt ( email )

Grüneburgplatz 1
Frankfurt am Main, 60323
Germany

Oliver Hinz

Goethe University Frankfurt - Faculty of Economics and Business Administration ( email )

Mertonstrasse 17-25
Frankfurt am Main, D-60325
Germany

Do you have a job opening that you would like to promote on SSRN?

Paper statistics

Downloads
221
Abstract Views
893
rank
196,121
PlumX Metrics