Ethical Risks of Autonomous Products: The Case of Mental Health Crises on AI Companion Applications

44 Pages Posted: 12 Sep 2022

See all articles by Julian De Freitas

Julian De Freitas

Harvard University - Business School (HBS)

Ahmet Kaan Uğuralp

Bilkent University

Zeliha Uğuralp

Bilkent University

Date Written: August 2022

Abstract

Increasingly, some products do not merely automate some piece of our lives but act as autonomous agents. When these technologies are not yet perfected, what are their risks? Here we explore the case of AI companion apps. Although these apps are designed for companionship rather than therapy, we use automated text analysis of human-AI conversations on these apps to find that consumers are nonetheless discussing mental health and find these discussions most engaging, increasing the chance that they will also consult these apps in times of crisis. Given this, we then submit mental health crisis messages to these apps and categorize whether the responses are appropriate—whether they recognize the crisis, and are empathetic, helpful, and provide a mental health resource. We find that most apps do indeed respond inappropriately, raising reputational and regulatory risks for brands, and welfare risks for consumers (e.g., encouraging them to harm themselves or others, or making them feel invalidated). These findings broaden our understanding of ethically relevant risks arising from the unconstrained nature of autonomous products.

Note:
Funding Information: Funding for this research was provided in part by Harvard Business School

Conflict of Interests: None to declare.

Keywords: Autonomy, Artificial Intelligence, Chatbots, New Technology, Brand Crises, Ethics, Mental Health, AI and Machine Learning, Well-being, Health, Applications and Software

Suggested Citation

De Freitas, Julian and Uğuralp, Ahmet Kaan and Uğuralp, Zeliha, Ethical Risks of Autonomous Products: The Case of Mental Health Crises on AI Companion Applications (August 2022). Harvard Business School Marketing Unit Working Paper No. 23-011, Harvard Business School Research Paper No. 23-011, Available at SSRN: https://ssrn.com/abstract=4188919 or http://dx.doi.org/10.2139/ssrn.4188919

Julian De Freitas (Contact Author)

Harvard University - Business School (HBS) ( email )

Soldiers Field Road
Morgan 270C
Boston, MA 02163
United States

Ahmet Kaan Uğuralp

Bilkent University

Bilkent, Ankara 06533
Turkey

Zeliha Uğuralp

Bilkent University

Bilkent, Ankara 06533
Turkey

Do you have a job opening that you would like to promote on SSRN?

Paper statistics

Downloads
25
Abstract Views
233
PlumX Metrics