Evaluating Metrics for Impact Quantification

Abstract

This project proposes concrete metrics to assess the human impact of machine learning applications, thus addressing the gap between ethics and quantitative measurement. Current discussions of AI ethics revolve around fairness, accountability, transparency, and explainability (the so-called “fate” principles), yet lack concrete metrics for practically implementing or measur- ing the ethical dimensions of AI. Our report proposes such metrics, defends their philosophical foundations, and illustrates how they can be implemented to facilitate analysis and decision making throughout an organization. We improve upon existing risk assessments of AI, moving us closer to the goal of precise, quantitative assessment of AI’s human impacts.

We outline a universal theory of human flourishing, based on Martha Nussbaum and Amartya Sen’s “capabilities approach.” This theory encom- passes broad categories such as environmental health, bodily health, and free- dom of affiliation. The approach’s wide scope and “ecumenical” nature allows us to circumvent contentious debates about what constitutes a good life while accommodating a broad array of reasonable views.

Next, we suggest selecting relevant capabilities that align with the goal of the domain wherein the AI model is deployed.

Last, we identify relevant metrics to measure an application’s impact on human flourishing and propose a “Human Impact Scorecard” that can include both qualitative and quantitative metrics. These scorecards allow for comparisons between applications, thus enabling informed decision-making. We illustrate this approach by applying it to three real-world case studies.

The report, up until this point, stands on its own. If a reader is curious about our methodology, in the latter portion of the report, we explore the philosoph- ical foundations, adjacent approaches, and previous work that informs our approach. This analysis reveals several philosophical challenges confronting multidimensional measures of human wellbeing, against which we establish requirements for a successful approach. We show how our approach satisfies these requirements better than any competing approach we know of.

Finally, we discuss potential extensions of this work, like accommodating a broader sociotechnical analysis of AI models and addressing other species of harm that might be caused by AI systems.

Keywords: artificial intelligence ethics, measurability, risk management, impact assessment