GPT-4’s Law School Grades: Con Law C, Crim C-, Law & Econ C, Partnership Tax B, Property B-, Tax B

Abstract

GPT-4 performs vastly better than ChatGPT or GPT-3.5 on legal tasks like the bar exam and statutory reasoning. To test GPT-4’s abilities, we ran it on our final exams this semester and graded its output alongside students’ exams. We found that it produced smoothly written answers that failed to spot many important issues, much like a bright student who had neither attended class often, nor thought deeply about the material. It uniformly performed below average – in every course. We provide observations that may help law professors detect students who cheat on exams using GPT-4.

Keywords: GPT-4, exam, artificial intelligence