AI Assistance in Legal Analysis: An Empirical Study

46 Pages Posted: 16 Aug 2023 Last revised: 23 Jan 2024

See all articles by Jonathan H. Choi

Jonathan H. Choi

University of Southern California

Daniel Schwarcz

University of Minnesota Law School

Date Written: August 13, 2023

Abstract

Can artificial intelligence (AI) augment human legal reasoning? To find out, we designed a novel experiment administering law school exams to students with and without access to GPT-4, the best-performing AI model currently available. We found that assistance from GPT-4 significantly enhanced performance on simple multiple-choice questions but not on complex essay questions. We also found that GPT-4’s impact depended heavily on the student’s starting skill level; students at the bottom of the class saw huge performance gains with AI assistance, while students at the top of the class saw performance declines. This suggests that AI may have an equalizing effect on the legal profession, mitigating inequalities between elite and nonelite lawyers. In addition, we graded exams written by GPT-4 alone to compare it with humans alone and AI-assisted humans. We found that GPT-4’s performance varied substantially depending on prompting methodology. With basic prompts, GPT-4 was a mediocre student, but with optimal prompting it outperformed both the average student and the average student with access to AI. This finding has important implications for the future of work, hinting that it may become advantageous to entirely remove humans from the loop for certain tasks.

Keywords: AI, ChatGPT, Legal Reasoning

Suggested Citation

Choi, Jonathan H. and Schwarcz, Daniel, AI Assistance in Legal Analysis: An Empirical Study (August 13, 2023). 73 Journal of Legal Education (forthcoming, 2024), Available at SSRN: https://ssrn.com/abstract=4539836 or http://dx.doi.org/10.2139/ssrn.4539836

Jonathan H. Choi

University of Southern California ( email )

2250 Alcazar Street
Los Angeles, CA 90089
United States

Daniel Schwarcz (Contact Author)

University of Minnesota Law School ( email )

229 19th Avenue South
Minneapolis, MN 55455
United States

HOME PAGE: http://www.law.umn.edu/profiles/daniel-schwarcz

Do you have negative results from your research you’d like to share?

Paper statistics

Downloads
3,504
Abstract Views
13,731
Rank
6,077
PlumX Metrics