Crowdsourcing Accurately and Robustly Predicts Supreme Court Decisions

11 Pages Posted: 13 Dec 2017  

Daniel Martin Katz

Illinois Tech - Chicago Kent College of Law; Stanford CodeX - The Center for Legal Informatics

Michael James Bommarito

LexPredict, LLC; Bommarito Consulting, LLC; Chicago-Kent College of Law - Illinois Institute of Technology; Michigan State College of Law

Josh Blackman

South Texas College of Law Houston

Date Written: December 11, 2017

Abstract

Scholars have increasingly investigated “crowdsourcing” as an alternative to expert-based judgment or purely data-driven approaches to predicting the future. Under certain conditions, scholars have found that crowd-sourcing can outperform these other approaches. However, despite interest in the topic and a series of successful use cases, relatively few studies have applied empirical model thinking to evaluate the accuracy and robustness of crowdsourcing in real-world contexts. In this paper, we offer three novel contributions. First, we explore a dataset of over 600,000 predictions from over 7,000 participants in a multi-year tournament to predict the decisions of the Supreme Court of the United States. Second, we develop a comprehensive crowd construction framework that allows for the formal description and application of crowdsourcing to real-world data. Third, we apply this framework to our data to construct more than 275,000 crowd models. We find that in out-of-sample historical simulations, crowdsourcing robustly outperforms the commonly-accepted null model, yielding the highest-known performance for this context at 80.8% case level accuracy. To our knowledge, this dataset and analysis represent one of the largest explorations of recurring human prediction to date, and our results provide additional empirical support for the use of crowdsourcing as a prediction method.

Keywords: judicial prediction, crowdsourcing, legal prediction, judicial crowdsourcing, legal analytics, legal data, model thinking, quantitative legal prediction

JEL Classification: K40, K41, H40

Suggested Citation

Katz, Daniel Martin and Bommarito, Michael James and Blackman, Josh, Crowdsourcing Accurately and Robustly Predicts Supreme Court Decisions (December 11, 2017). Available at SSRN: https://ssrn.com/abstract=3085710 or http://dx.doi.org/10.2139/ssrn.3085710

Daniel Martin Katz (Contact Author)

Illinois Tech - Chicago Kent College of Law ( email )

565 W. Adams St.
Chicago, IL 60661-3691
United States

HOME PAGE: http://www.danielmartinkatz.com/

Stanford CodeX - The Center for Legal Informatics ( email )

559 Nathan Abbott Way
Stanford, CA 94305-8610
United States

HOME PAGE: http://law.stanford.edu/directory/daniel-katz/

Michael James Bommarito

LexPredict, LLC ( email )

MI
United States

HOME PAGE: http://lexpredict.com

Bommarito Consulting, LLC ( email )

MI 48098
United States

HOME PAGE: http://bommaritollc.com

Chicago-Kent College of Law - Illinois Institute of Technology ( email )

565 W. Adams St.
Chicago, IL 60661-3691
United States

Michigan State College of Law ( email )

318 Law College Building
East Lansing, MI 48824-1300
United States

Josh Blackman

South Texas College of Law Houston ( email )

1303 San Jacinto Street
Houston, TX 77002
United States

Register to save articles to
your library

Register

Paper statistics

Downloads
241
rank
113,921
Abstract Views
1,116
PlumX