Caveat Lector: Large Language Models in Legal Practice

60 Pages Posted: 15 Mar 2024

See all articles by Eliza Mik

Eliza Mik

The Chinese University of Hong Kong (CUHK) - Faculty of Law; TILT; Melbourne Law School

Date Written: February 16, 2024

Abstract

Reader Beware. The current fascination with large language models, or “LLMs,” derives from the fact that many users lack the expertise to evaluate the quality of the generated text. LLMs may therefore appear more capable than they actually are. The dangerous combination of fluency and superficial plausibility leads to the temptation to trust the generated text and creates the risk of overreliance. Who wouldn’t trust perfect legalese?

Drawing from recent findings in both technical and legal scholarship, this Article counterbalances the overly optimistic predictions as to the role of LLMs in legal practice. Integrating LLMs into legal workstreams without a better comprehension of their limitations, will create inefficiencies if not outright risks. Notwithstanding their unprecedented ability to generate text, LLMs do not understand text. Without the ability to understand the meaning of words, LLMs will remain unable to use language, to acquire knowledge and to perform complex reasoning tasks.

Trained to model language on the basis of stochastic word predictions, LLMs cannot distinguish fact from fiction. Their “knowledge” of the law is limited to word strings memorized in their parameters. It is also often incomplete and largely incorrect. LLMs operate at the level of word distributions, not at the level of verified facts. The resulting propensity to hallucinate, to produce statements that are incorrect but appear helpful and relevant, is alarming in high-risk areas like legal services. For the time being, lawyers should beware of relying on text generated by LLMs.

Keywords: legal technology, legal practice, Large Language Models, Computational Law

JEL Classification: K10, K20, K30, K40

Suggested Citation

Mik, Eliza, Caveat Lector: Large Language Models in Legal Practice (February 16, 2024). The Chinese University of Hong Kong Faculty of Law Research Paper No. 2024-04, Volume 19 Rutgers Bus. L.R. 2 2024 (forthcoming) , Available at SSRN: https://ssrn.com/abstract=4757452

Eliza Mik (Contact Author)

The Chinese University of Hong Kong (CUHK) - Faculty of Law ( email )

6/F, Lee Shau Kee Building
Shatin, New Territories
Kowloon, Sha Tin
Hong Kong

TILT ( email )

P.O.Box 90153
Prof. Cobbenhagenlaan 221
Tilburg, 5037
Netherlands

Melbourne Law School ( email )

185 Pelham Street
Melbourne, VIC 3010
Australia

Do you have a job opening that you would like to promote on SSRN?

Paper statistics

Downloads
353
Abstract Views
1,029
Rank
159,171
PlumX Metrics