ChatGPT Goes to Law School

16 Pages Posted: 25 Jan 2023 Last revised: 1 Feb 2023

See all articles by Jonathan H. Choi

Jonathan H. Choi

University of Minnesota Law School

Kristin E. Hickman

University of Minnesota - Twin Cities - School of Law

Amy Monahan

University of Minnesota Law School

Daniel Schwarcz

University of Minnesota Law School

Date Written: January 23, 2023

Abstract

How well can AI models write law school exams without human assistance? To find out, we used the widely publicized AI model ChatGPT to generate answers on four real exams at the University of Minnesota Law School. We then blindly graded these exams as part of our regular grading processes for each class. Over 95 multiple choice questions and 12 essay questions, ChatGPT performed on average at the level of a C+ student, achieving a low but passing grade in all four courses. After detailing these results, we discuss their implications for legal education and lawyering. We also provide example prompts and advice on how ChatGPT can assist with legal writing.

Keywords: ChatGPT, law school, AI, natural language processing, Legal Data, NLP, Legal NLP, Legal Analytics, natural language understanding, evaluation, machine learning, artificial intelligence, artificial intelligence and law

Suggested Citation

Choi, Jonathan H. and Hickman, Kristin E. and Monahan, Amy and Schwarcz, Daniel B., ChatGPT Goes to Law School (January 23, 2023). Minnesota Legal Studies Research Paper No. 23-03, Available at SSRN: https://ssrn.com/abstract=4335905 or http://dx.doi.org/10.2139/ssrn.4335905

Jonathan H. Choi (Contact Author)

University of Minnesota Law School ( email )

229 19th Avenue South
Minneapolis, MN 55455
United States

Kristin E. Hickman

University of Minnesota - Twin Cities - School of Law ( email )

229 19th Avenue South
Minneapolis, MN 55455
United States
612-624-2915 (Phone)

Amy Monahan

University of Minnesota Law School ( email )

229 19th Avenue South
Minneapolis, MN 55455
United States

Daniel B. Schwarcz

University of Minnesota Law School ( email )

229 19th Avenue South
Minneapolis, MN 55455
United States

HOME PAGE: http://www.law.umn.edu/profiles/daniel-schwarcz

Do you have a job opening that you would like to promote on SSRN?

Paper statistics

Downloads
5,093
Abstract Views
23,858
Rank
2,646
PlumX Metrics