Evaluating Metrics for Impact Quantification

Jenkins, Ryan and Lorenzo Nericcio. “Exploring Methods for Impact Quantification.” Report funded by Center for Advancing Safety of Machine Intelligence (casmi). July, 2023

69 Pages Posted: 17 Nov 2023

See all articles by Ryan Jenkins

Ryan Jenkins

California Polytechnic State University

Lorenzo Nericcio

San Diego State University - College of Arts and Letters

Date Written: June 24, 2023

Abstract

This project proposes concrete metrics to assess the human impact of machine learning applications, thus addressing the gap between ethics and quantitative measurement. Current discussions of AI ethics revolve around fairness, accountability, transparency, and explainability (the so-called “fate” principles), yet lack concrete metrics for practically implementing or measur- ing the ethical dimensions of AI. Our report proposes such metrics, defends their philosophical foundations, and illustrates how they can be implemented to facilitate analysis and decision making throughout an organization. We improve upon existing risk assessments of AI, moving us closer to the goal of precise, quantitative assessment of AI’s human impacts.

We outline a universal theory of human flourishing, based on Martha Nussbaum and Amartya Sen’s “capabilities approach.” This theory encom- passes broad categories such as environmental health, bodily health, and free- dom of affiliation. The approach’s wide scope and “ecumenical” nature allows us to circumvent contentious debates about what constitutes a good life while accommodating a broad array of reasonable views.

Next, we suggest selecting relevant capabilities that align with the goal of the domain wherein the AI model is deployed.

Last, we identify relevant metrics to measure an application’s impact on human flourishing and propose a “Human Impact Scorecard” that can include both qualitative and quantitative metrics. These scorecards allow for comparisons between applications, thus enabling informed decision-making. We illustrate this approach by applying it to three real-world case studies.

The report, up until this point, stands on its own. If a reader is curious about our methodology, in the latter portion of the report, we explore the philosoph- ical foundations, adjacent approaches, and previous work that informs our approach. This analysis reveals several philosophical challenges confronting multidimensional measures of human wellbeing, against which we establish requirements for a successful approach. We show how our approach satisfies these requirements better than any competing approach we know of.

Finally, we discuss potential extensions of this work, like accommodating a broader sociotechnical analysis of AI models and addressing other species of harm that might be caused by AI systems.

Keywords: artificial intelligence ethics, measurability, risk management, impact assessment

Suggested Citation

Jenkins, Ryan and Nericcio, Lorenzo, Evaluating Metrics for Impact Quantification (June 24, 2023). Jenkins, Ryan and Lorenzo Nericcio. “Exploring Methods for Impact Quantification.” Report funded by Center for Advancing Safety of Machine Intelligence (casmi). July, 2023, Available at SSRN: https://ssrn.com/abstract=4607297 or http://dx.doi.org/10.2139/ssrn.4607297

Ryan Jenkins (Contact Author)

California Polytechnic State University ( email )

San Luis Obispo, CA 93407
United States

Lorenzo Nericcio

San Diego State University - College of Arts and Letters ( email )

United States

Do you have negative results from your research you’d like to share?

Paper statistics

Downloads
42
Abstract Views
191
PlumX Metrics