Re-Evaluating GPT-4's Bar Exam Performance

15 Pages Posted: 18 May 2023 Last revised: 26 Sep 2023

See all articles by Eric Martínez

Eric Martínez

Massachusetts Institute of Technology (MIT)

Date Written: May 8, 2023


Perhaps the most widely touted of GPT-4's at-launch, zero-shot capabilities has been its reported 90th-percentile performance on the Uniform Bar Exam, with its reported 80-percentile-points boost over its predecessor, GPT-3.5, far exceeding that for any other exam. This paper investigates the methodological challenges in documenting and verifying the 90th-percentile claim, presenting four sets of findings that suggest that OpenAI's estimates of GPT-4's UBE percentile, though clearly an impressive leap over those of GPT-3.5, appear to be overinflated, particularly if taken as a “conservative” estimate representing “the lower range of percentiles,” and more so if meant to reflect the actual capabilities of a practicing lawyer.

First, although GPT-4's UBE score nears the 90th percentile when examining approximate conversions from February administrations of the Illinois Bar Exam, these estimates are heavily skewed towards repeat test-takers who failed the July administration and score significantly lower than the general test-taking population. Second, data from a recent July administration of the same exam suggests GPT-4's overall UBE percentile was ~68th percentile, and ~48th percentile on essays. Third, examining official NCBE data and using several conservative statistical assumptions, GPT-4's performance against first-time test takers is estimated to be ~63rd percentile, including ~42nd percentile on essays. Fourth, when examining only those who passed the exam (i.e. licensed or license-pending attorneys), GPT-4's performance is estimated to drop to ~48th percentile overall, and ~15th percentile on essays.

Taken together, these findings carry timely insights for the desirability and feasibility of outsourcing legally relevant tasks to AI models, as well as for the importance for AI developers to implement rigorous and transparent capabilities evaluations to help secure safe and trustworthy AI.

Keywords: GPT, GPT-4, ChatGPT, Bar Exam, NLP, Legal NLP, Legal Analytics, natural language processing, natural language understanding, machine learning, artificial intelligence, artificial intelligence and law, Neural NLP, Legal Tech

JEL Classification: C45, C55, K49, O33, O30

Suggested Citation

Martínez, Eric, Re-Evaluating GPT-4's Bar Exam Performance (May 8, 2023). LPP Working Paper No. 2-2023, Available at SSRN: or

Eric Martínez (Contact Author)

Massachusetts Institute of Technology (MIT) ( email )

Do you have negative results from your research you’d like to share?

Paper statistics

Abstract Views
PlumX Metrics