Misplaced Trust in Artificial Professional Advice

Abstract

The professional relationship constitutes a distinctive social interaction in which the professional has knowledge that the client or patient lacks but needs to make important decisions. A specific legal and ethical framework governs this relationship. Traditionally, this framework assumes interactions between human actors and is designed to safeguard normative values inherent to human-centered professional advice-giving. Prominent among the foundational assumptions are expectations of competence, loyalty, and trust. Individuals seeking professional advice expect to receive accurate and comprehensive advice reflecting the insights of the profession. 

This Essay examines the social relationship based on knowledge asymmetries and trust underlying the human professional relationship and what its absence means for the output of AI-based non-human actors-including generative AI systems like large language models (LLMs)-dispensing what looks like professional advice. As I will argue in this Essay, the trust problem is twofold: first, trust in AI output resembling professional advice is misplaced; second, untrustworthy AI displacing professionals in turn undermines trust in human professionals.