The Hidden Assumptions Behind Counterfactual Explanations and Principal Reasons

ACM Conference on Fairness, Accountability, and Transparency (FAT*), 2020

17 Pages Posted: 9 Jan 2020

See all articles by Solon Barocas

Solon Barocas

Microsoft Research; Cornell University

Andrew D. Selbst

UCLA School of Law

Manish Raghavan

Massachusetts Institute of Technology (MIT)

Date Written: December 12, 2019


Counterfactual explanations are gaining prominence within technical, legal, and business circles as a way to explain the decisions of a machine learning model. These explanations share a trait with the long-established "principal reason" explanations required by U.S. credit laws: they both explain a decision by highlighting a set of features deemed most relevant – and withholding others.

These "feature-highlighting explanations" have several desirable properties: They place no constraints on model complexity, do not require model disclosure, detail what needed to be different to achieve a different decision, and seem to automate compliance with the law. But they are far more complex and subjective than they appear.

In this paper, we demonstrate that the utility of feature-highlighting explanations relies on a number of easily overlooked assumptions: that the recommended change in feature values clearly maps to real-world actions, that features can be made commensurate by looking only at the distribution of the training data, that features are only relevant to the decision at hand, and that the underlying model is stable over time, monotonic, and limited to binary outcomes.

We then explore several consequences of acknowledging and attempting to address these assumptions, including a paradox in the way that feature-highlighting explanations aim to respect autonomy, the unchecked power that feature-highlighting explanations grant decision makers, and a tension between making these explanations useful and the need to keep the model hidden.

While new research suggests several ways that feature-highlighting explanations can work around some of the problems that we identify, the disconnect between features in the model and actions in the real world – and the subjective choices necessary to compensate for this – must be understood before these techniques can be usefully implemented.

Keywords: interpretability, counterfactual explanation, principal reasons, accountability, autonomy

JEL Classification: G28

Suggested Citation

Barocas, Solon and Selbst, Andrew D. and Raghavan, Manish, The Hidden Assumptions Behind Counterfactual Explanations and Principal Reasons (December 12, 2019). ACM Conference on Fairness, Accountability, and Transparency (FAT*), 2020, Available at SSRN:

Solon Barocas

Microsoft Research

300 Lafayette Street
New York, NY 10012
United States

Cornell University ( email )

Ithaca, NY 14853
United States

Manish Raghavan (Contact Author)

Massachusetts Institute of Technology (MIT) ( email )

77 Massachusetts Avenue
50 Memorial Drive
Cambridge, MA 02139-4307
United States

Do you have a job opening that you would like to promote on SSRN?

Paper statistics

Abstract Views
PlumX Metrics