Moral Crumple Zones: Cautionary Tales in Human-Robot Interaction (pre-print)

Engaging Science, Technology, and Society (pre-print)

29 Pages Posted: 3 Apr 2016 Last revised: 15 Mar 2019

See all articles by Madeleine Clare Elish

Madeleine Clare Elish

Google Inc.; University of Oxford - Oxford Internet Institute

Date Written: March 1, 2019

Abstract

As debates about the policy and ethical implications of AI systems grow, it will be increasingly important to accurately locate who is responsible when agency is distributed in a system and control over an action is mediated through time and space. Analyzing several high-profile accidents involving complex and automated socio-technical systems and the media coverage that surrounded them, I introduce the concept of a moral crumple zone to describe how responsibility for an action may be misattributed to a human actor who had limited control over the behavior of an automated or autonomous system. Just as the crumple zone in a car is designed to absorb the force of impact in a crash, the human in a highly complex and automated system may become simply a component—accidentally or intentionally—that bears the brunt of the moral and legal responsibilities when the overall system malfunctions. While the crumple zone in a car is meant to protect the human driver, the moral crumple zone protects the integrity of the technological system, at the expense of the nearest human operator. The concept is both a challenge to and an opportunity for the design and regulation of human-robot systems. At stake in articulating moral crumple zones is not only the misattribution of responsibility but also the ways in which new forms of consumer and worker harm may develop in new complex, automated, or purported autonomous technologies.

Keywords: HRI, ethics, driverless cars, human factors, automation, autonomy, human in the loop, HCI,autonomous vehicles, responsibility, machine learning, human factors, accidents, social perceptions of technology, self-driving cars, robots, human-in-the-loop, human-robot interaction

Suggested Citation

Elish, Madeleine Clare, Moral Crumple Zones: Cautionary Tales in Human-Robot Interaction (pre-print) (March 1, 2019). Engaging Science, Technology, and Society (pre-print), Available at SSRN: https://ssrn.com/abstract=2757236 or http://dx.doi.org/10.2139/ssrn.2757236

Madeleine Clare Elish (Contact Author)

Google Inc. ( email )

1600 Amphitheatre Parkway
Second Floor
Mountain View, CA 94043
United States

University of Oxford - Oxford Internet Institute ( email )

1 St. Giles
University of Oxford
Oxford OX1 3PG Oxfordshire, Oxfordshire OX1 3JS
United Kingdom

Do you have negative results from your research you’d like to share?

Paper statistics

Downloads
7,328
Abstract Views
55,912
Rank
1,748
PlumX Metrics