Topic Modeling the President: Conventional and Computational Methods

70 Pages Posted: 12 Dec 2017 Last revised: 17 Sep 2018

J. B. Ruhl

Vanderbilt University - Law School

John Nay

New York University School of Law; Harvard University - Berkman Klein Center for Internet & Society

Jonathan M. Gilligan

Vanderbilt University - Department of Earth and Environmental Sciences

Date Written: December 11, 2017

Abstract

Law is generally represented through text, and lawyers have for centuries classified large bodies of legal text into distinct topics — they “topic model” the law. But large bodies of legal documents present challenges for conventional topic modeling methods. The task of gathering, reviewing, coding, sorting, and assessing a body of tens of thousands of legal documents is a daunting proposition. Recent advances in computational text analytics, a subset of the field of “artificial intelligence,” are already gaining traction in legal practice settings such as e-discovery by leveraging the speed and capacity of computers to process enormous bodies of documents. Differences between conventional and computational methods, however, suggest that computational text modeling has its own limitations, but that the two methods used in unison could be a powerful research tool for legal scholars in their research as well.

To explore that potential — and to do so critically rather than under the “shiny rock” spell of artificial intelligence — we assembled a large corpus of presidential documents to assess how computational topic modeling compares to conventional methods and evaluate how legal scholars can best make use of the computational methods. The presidential documents of interest comprise presidential “direct actions,” such as executive orders, presidential memoranda, proclamations, and other exercises of authority the president can take alone, without congressional concurrence or agency involvement. Presidents have been issuing direct actions throughout the history of the republic, and while they have often been the target of criticism and controversy in the past, lately they have become a tinderbox of debate. Hence, although long ignored by political scientists and legal scholars, there has been a surge of interest in the scope, content, and impact of presidential direct actions.

Legal and policy scholars modeling direct actions into substantive topic classifications thus far have not employed computational methods. This gives us an opportunity to compare results of the two methods. We generated computational topic models of all direct actions over time periods other scholars have studied using conventional methods, and did the same for a case study of environmental policy direct actions. Our computational model of all direct actions closely matched one of the two comprehensive empirical models developed using conventional methods. By contrast, our environmental case study model differed markedly from the only other empirical topic model of environmental policy direct actions, revealing that the conventional methods model included trivial categories and omitted important alternative topics.

Our findings support the assessment that computational topic modeling, provided a sufficiently large corpus of documents is used, can provide important insights for legal scholars in designing and validating their topic models of legal text. To be sure, computational topic modeling used alone has its limitations, some of which are evident in our models, but when used along with conventional methods, it opens doors towards reaching more confident conclusions about how to conceptualize topics in law. Drawing from these results, we offer several use cases for computational topic modeling in legal research. At the front-end, researchers can use the method to generate better and more complete model hypotheses. At the back-end, the method can effectively be used, as we did, to validate existing topic models. And at a meta-scale, the method opens windows to test and challenge conventional legal theory. Legal scholars can do all of these without “the machines,” but there is good reason to believe we can do it better with them in the toolkit.

Suggested Citation

Ruhl, J. B. and Nay, John and Gilligan, Jonathan M., Topic Modeling the President: Conventional and Computational Methods (December 11, 2017). George Washington Law Review, Forthcoming; Vanderbilt Law Research Paper No. 17-62. Available at SSRN: https://ssrn.com/abstract=3086226

J. B. Ruhl (Contact Author)

Vanderbilt University - Law School ( email )

131 21st Avenue South
Nashville, TN 37203-1181
United States

John Nay

New York University School of Law

40 Washington Square South
New York, NY 10012-1099
United States

Harvard University - Berkman Klein Center for Internet & Society ( email )

Harvard Law School
23 Everett, 2nd Floor
Cambridge, MA 02138
United States

Jonathan M. Gilligan

Vanderbilt University - Department of Earth and Environmental Sciences ( email )

VU Station B #351805
2301 Vanderbilt Place
Nashville, TN 37235-1805
United States
615.322.2420 (Phone)
615.322.2138 (Fax)

HOME PAGE: http://my.vanderbilt.edu/jonathangilligan

Register to save articles to
your library

Register

Paper statistics

Downloads
86
rank
272,894
Abstract Views
490
PlumX