Against Predictive Optimization: On the Legitimacy of Decision-Making Algorithms that Optimize Predictive Accuracy

45 Pages Posted: 18 Oct 2022 Last revised: 17 Feb 2023

See all articles by Angelina Wang

Angelina Wang

Princeton University

Sayash Kapoor

Princeton University

Solon Barocas

Microsoft Research; Cornell University

Arvind Narayanan

Princeton University

Date Written: October 4, 2022

Abstract

We formalize predictive optimization, a category of decision-making algorithms that use machine learning (ML) to predict future outcomes of interest about individuals. For example, pre-trial risk prediction algorithms such as COMPAS use ML to predict whether an individual will re-offend in the future. Our thesis is that predictive optimization raises a distinctive and serious set of normative concerns that render it presumptively illegitimate. To test this, we review 387 reports, articles, and web pages from academia, industry, non-profits, governments, and modeling contests, and find many real-world examples of predictive optimization. We select eight particularly consequential examples as case studies. Simultaneously, we develop a set of normative and technical critiques that challenge the claims made by the developers of these applications—in particular, claims of increased accuracy, efficiency, and fairness. Our key finding is that these critiques apply to each of the applications, are not easily evaded by redesigning the systems, and thus challenge the legitimacy of their deployment. We argue that the burden of evidence for justifying why the deployment of predictive optimization is not harmful should rest with the developers of the tools. Based on our analysis, we provide a rubric of critical questions that can be used to deliberate or contest the legitimacy of specific predictive optimization applications.

Suggested Citation

Wang, Angelina and Kapoor, Sayash and Barocas, Solon and Narayanan, Arvind, Against Predictive Optimization: On the Legitimacy of Decision-Making Algorithms that Optimize Predictive Accuracy (October 4, 2022). Available at SSRN: https://ssrn.com/abstract=4238015

Angelina Wang

Princeton University ( email )

Sayash Kapoor

Princeton University ( email )

United States

Solon Barocas (Contact Author)

Microsoft Research

300 Lafayette Street
New York, NY 10012
United States

Cornell University ( email )

Ithaca, NY 14853
United States

Arvind Narayanan

Princeton University ( email )

Princeton, NJ 08540
United States

Do you have negative results from your research you’d like to share?

Paper statistics

Downloads
2,557
Abstract Views
6,803
Rank
10,082
PlumX Metrics