Strange Loops: Apparent versus Actual Human Involvement in Automated Decision-Making

27 Pages Posted: 14 Oct 2019

See all articles by Kiel Brennan-Marquez

Kiel Brennan-Marquez

University of Connecticut - School of Law

Daniel Susser

Penn State

Karen Levy

Cornell University

Date Written: October 2, 2019

Abstract

The era of AI-based decision-making fast approaches, and anxiety is mounting about when, and why, we should keep “humans in the loop” (“HITL”). Thus far, commentary has focused primarily on two questions: whether, and when, keeping humans involved will improve the results of decision-making (making them safer or more accurate), and whether, and when, non-accuracy-related values—legitimacy, dignity, and so forth—are vindicated by the inclusion of humans in decision-making. Here, we take up a related but distinct question, which has eluded the scholarship thus far: does it matter if humans appear to be in the loop of decision-making, independent from whether they actually are? In other words, what is stake in the disjunction between whether humans in fact have ultimate authority over decision-making versus whether humans merely seem, from the outside, to have such authority?

Our argument proceeds in four parts. First, we build our formal model, enriching the HITL question to include not only whether humans are actually in the loop of decision-making, but also whether they appear to be so. Second, we describe situations in which the actuality and appearance of HITL align: those that seem to involve human judgment and actually do, and those that seem automated and actually are. Third, we explore instances of misalignment: situations in which systems that seem to involve human judgment actually do not, and situations in which systems that hold themselves out as automated actually rely on humans operating “behind the curtain.” Fourth, we examine the normative issues that result from HITL misalignment, arguing that it challenges individual decision-making about automated systems and complicates collective governance of automation.

JEL Classification: K10

Suggested Citation

Brennan-Marquez, Kiel and Susser, Daniel and Levy, Karen, Strange Loops: Apparent versus Actual Human Involvement in Automated Decision-Making (October 2, 2019). Berkeley Technology Law Journal, Forthcoming. Available at SSRN: https://ssrn.com/abstract=3462901

Kiel Brennan-Marquez (Contact Author)

University of Connecticut - School of Law ( email )

65 Elizabeth Street
Hartford, CT 06105
United States

Daniel Susser

Penn State ( email )

E325 Westgate Building
University Park, PA 16802
United States

HOME PAGE: http://www.danielsusser.info

Karen Levy

Cornell University ( email )

Ithaca, NY 14853
United States

Register to save articles to
your library

Register

Paper statistics

Downloads
213
Abstract Views
1,317
rank
145,366
PlumX Metrics