Model Multiplicity: Opportunities, Concerns, and Solutions

29 Pages Posted: 28 Jun 2022 Last revised: 14 Feb 2023

See all articles by Emily Black

Emily Black

Columbia University - Barnard College

Manish Raghavan

Harvard University - John A. Paulson School of Engineering and Applied Sciences

Solon Barocas

Microsoft Research; Cornell University

Date Written: January 21, 2022

Abstract

Recent scholarship has brought attention to the fact that there often exist multiple models for a given prediction task with equal accuracy that differ in their individual-level predictions or aggregate properties. This phenomenon---which we call model multiplicity---can introduce a good deal of flexibility into the model selection process, creating a range of exciting opportunities. By demonstrating that there are many different ways of making equally accurate predictions, multiplicity gives model developers the freedom to prioritize other values in their model selection process without having to abandon their commitment to maximizing accuracy. However, multiplicity also brings to light a concerning truth: model selection on the basis of accuracy alone---the default procedure in many deployment scenarios---fails to consider what might be meaningful differences between equally accurate models with respect to other criteria such as fairness, robustness, and interpretability. Unless these criteria are taken into account explicitly, developers might end up making unnecessary trade-offs or could even mask intentional discrimination. Furthermore, the prospect that there might exist another model of equal accuracy that flips a prediction for a particular individual may lead to a crisis in justifiability: why should an individual be subject to an adverse model outcome if there exists an equally accurate model that treats them more favorably? In this work, we investigate how to take advantage of the flexibility afforded by model multiplicity while addressing the concerns with justifiability that it might raise.

Keywords: Model multiplicity, predictive multiplicity, procedural multiplicity, fairness, discrimination, recourse, arbitrariness

Suggested Citation

Black, Emily and Raghavan, Manish and Barocas, Solon, Model Multiplicity: Opportunities, Concerns, and Solutions (January 21, 2022). Available at SSRN: https://ssrn.com/abstract=4142472 or http://dx.doi.org/10.2139/ssrn.4142472

Emily Black (Contact Author)

Columbia University - Barnard College ( email )

3009 Broadway
New York, NY 10027
United States

Manish Raghavan

Harvard University - John A. Paulson School of Engineering and Applied Sciences ( email )

150 Western Ave
Boston, MA 02134
United States

Solon Barocas

Microsoft Research

300 Lafayette Street
New York, NY 10012
United States

Cornell University ( email )

Ithaca, NY 14853
United States

Do you have a job opening that you would like to promote on SSRN?

Paper statistics

Downloads
91
Abstract Views
545
Rank
538,520
PlumX Metrics