An Uncommon Task: Participatory Design in Legal AI

Fernando Delgado, Solon Barocas, & Karen Levy. 2022. An Uncommon Task: Participatory Design in Legal AI. In Proceedings of the ACM on Human-Computer Interaction, 6, CSCW1, Article 51 (April 2022), 23 pages, https://doi.org/10.1145/3512898

23 Pages Posted: 1 Apr 2022

See all articles by Fernando Delgado

Fernando Delgado

Cornell University

Solon Barocas

Microsoft Research; Cornell University

Karen Levy

Cornell University

Date Written: April 1, 2022

Abstract

Despite growing calls for participation in AI design, there are to date few empirical studies of what these processes look like and how they can be structured for meaningful engagement with domain experts. In this paper, we examine a notable yet understudied AI design process in the legal domain that took place over a decade ago, the impact of which still informs legal automation efforts today. Specifically, we examine the design and evaluation activities that took place from 2006 to 2011 within the Text REtrieval Conference’s (TREC) Legal Track, a computational research venue hosted by the National Institute of Standards and Technologies. The Legal Track of TREC is notable in the history of AI research and practice because it relied on a range of participatory approaches to facilitate the design and evaluation of new computational techniques—in this case, for automating attorney document review for civil litigation matters. Drawing on archival research and interviews with coordinators of the Legal Track of TREC, our analysis reveals how an interactive simulation methodology allowed computer scientists and lawyers to become co-designers and helped bridge the chasm between computational research and real-world, high-stakes litigation practice. In analyzing this case from the recent past, our aim is to empirically ground contemporary critiques of AI development and evaluation and the calls for greater participation as a means to address them.

Keywords: legal technology, participatory design, artificial intelligence, electronic discovery, civil procedure, technology assisted review

Suggested Citation

Delgado, Fernando and Barocas, Solon and Barocas, Solon and Levy, Karen, An Uncommon Task: Participatory Design in Legal AI (April 1, 2022). Fernando Delgado, Solon Barocas, & Karen Levy. 2022. An Uncommon Task: Participatory Design in Legal AI. In Proceedings of the ACM on Human-Computer Interaction, 6, CSCW1, Article 51 (April 2022), 23 pages, https://doi.org/10.1145/3512898, Available at SSRN: https://ssrn.com/abstract=4058429

Fernando Delgado (Contact Author)

Cornell University ( email )

Ithaca, NY
United States

Solon Barocas

Microsoft Research

641 Avenue of Americas
New York, NY 10011
United States

Cornell University ( email )

Ithaca, NY 14853
United States

Karen Levy

Cornell University ( email )

Ithaca, NY 14853
United States

Do you have a job opening that you would like to promote on SSRN?

Paper statistics

Downloads
14
Abstract Views
148
PlumX Metrics