Legal Scholarship Through the Lens of Generative AI, Darkly

Abstract

Unlike traditional search engines limited to connecting users to original source content, generative AI systems produce new, ad-hoc sources of information derived primarily from patterns in their training data and information fed into the system as context. As such, generative AI systems can play a mediating role between users and information sources, especially when these systems are integrated into databases and web search engines. This article examines how GPT-4 (ChatGPT) interacts with law review articles, revealing its unreliability in summarizing them independently but notable accuracy when provided with full-text input. Retrieval augmented generation (RAG) offers a potential solution for improving AI accuracy in a more automated way, yet concerns persist about algorithmic bias, authors' rights, and the impact on legal scholarship. Law librarians must carefully consider these factors when determining how their institutions' scholarly work is accessed and used by AI systems.