LLM vs. Lawyers: Identifying a Subset of Summary Judgments in a Large UK Case Law Dataset

37 Pages Posted: 13 Mar 2024 Last revised: 25 Mar 2024

See all articles by Ahmed Izzidien

Ahmed Izzidien

University of Cambridge

Holli Sargeant

University of Cambridge, Faculty of Law

Felix Steffek

University of Cambridge - Faculty of Law

Date Written: March 3, 2024

Abstract

To undertake computational research of the law, efficiently identifying datasets of court decisions that relate to a specific legal issue is a crucial yet challenging endeavour. This study addresses the gap in the literature working with large legal corpora about how to isolate cases, in our case summary judgments, from a large corpus of UK court decisions. We introduce a comparative analysis of two computational methods: (1) a traditional natural language processing-based approach leveraging expert-generated keywords and logical operators and (2) an innovative application of the Claude 2 large language model to classify cases based on content-specific prompts. We use the Cambridge Law Corpus of 356,011 UK court decisions and determine that the large language model achieves a weighted F1 score of 0.94 versus 0.78 for keywords. Despite iterative refinement, the search logic based on keywords fails to capture nuances in legal language. We identify and extract 3,102 summary judgment cases, enabling us to map their distribution across various UK courts over a temporal span. The paper marks a pioneering step in employing advanced natural language processing to tackle core legal research tasks, demonstrating how these technologies can bridge systemic gaps and enhance the accessibility of legal information. We share the extracted dataset metrics to support further research on summary judgments.

Keywords: large language models, computational legal methods, legal corpora, UK case law, LegalAI, summary judgment, regular expression

Suggested Citation

Izzidien, Ahmed and Sargeant, Holli and Steffek, Felix, LLM vs. Lawyers: Identifying a Subset of Summary Judgments in a Large UK Case Law Dataset (March 3, 2024). University of Cambridge Faculty of Law Research Paper No. 10/2024, Available at SSRN: https://ssrn.com/abstract=4746305 or http://dx.doi.org/10.2139/ssrn.4746305

Ahmed Izzidien

University of Cambridge ( email )

10 West Road
Cambridge, CB3 9DZ
United Kingdom

Holli Sargeant (Contact Author)

University of Cambridge, Faculty of Law ( email )

Felix Steffek

University of Cambridge - Faculty of Law ( email )

10 West Road
Cambridge, CB3 9DZ
United Kingdom

HOME PAGE: http://www.law.cam.ac.uk/people/academic/f-steffek/6136

Do you have a job opening that you would like to promote on SSRN?

Paper statistics

Downloads
122
Abstract Views
363
Rank
440,846
PlumX Metrics