Credibility in Empirical Legal Analysis

87 Brooklyn Law Review (forthcoming 2022)

SMU Dedman School of Law Legal Studies Research Paper No. 434

49 Pages Posted: 8 Aug 2019 Last revised: 20 Apr 2021

See all articles by Hillel J. Bavli

Hillel J. Bavli

Southern Methodist University - Dedman School of Law

Date Written: August 7, 2019

Abstract

Quantitative empirical research is central to both legal scholarship and litigation, but there is little confidence in it. This is because researchers can manipulate data to arrive at any result they seek to find. The root of the problem is data fishing, the practice of using data to search for and selectively report results that are statistically significant or otherwise favorable to the researcher. For reasons explained in this article, data fishing invalidates statistical results by causing false positives and false impressions. It creates an environment in which, at best, readers are highly skeptical of statistical analysis and, at worst, they base important decisions, such as policy decisions and jury verdicts, on incorrect information. The practice is nevertheless prevalent in law—often committed by well-intentioned researchers who are unaware of its harms or unaware that their analysis constitutes data fishing.

This article exposes the harm that data fishing in empirical legal research causes. It then develops a framework for eliminating data fishing and restoring confidence in empirical analysis in legal scholarship and litigation. This framework, which I call DASS (an acronym for Design, Analyze, Scrutinize, and Substantiate), builds on methods in statistics and is designed for researchers to use to safeguard against data fishing and for consumers of empirical research—including scholars, courts, policymakers, and members of the public—to use to evaluate the reliability of a researcher’s statistical claims. DASS is designed to be simple and flexible, tailored to suit empirical research in law, and a substantial advancement over current anti-data-fishing practices in the social sciences, which have generally been ineffective. It can be applied broadly as a framework for credibility in empirical legal research, as well as to address a range of classical challenges in litigation, such as the hired-gun and battle-of-the-experts problems in evidence law.

Keywords: empirical legal studies, data fishing, reliability, accuracy, law and statistics, quantitative analysis, methodology, research practices, expert evidence, empirical analysis, hired gun, false positives, battle of the experts, statistical analysis, data manipulation, scientific evidence

Suggested Citation

Bavli, Hillel, Credibility in Empirical Legal Analysis (August 7, 2019). 87 Brooklyn Law Review (forthcoming 2022), SMU Dedman School of Law Legal Studies Research Paper No. 434, Available at SSRN: https://ssrn.com/abstract=3434095 or http://dx.doi.org/10.2139/ssrn.3434095

Hillel Bavli (Contact Author)

Southern Methodist University - Dedman School of Law ( email )

P.O. Box 750116
Dallas, TX 75275
United States

Do you have a job opening that you would like to promote on SSRN?

Paper statistics

Downloads
131
Abstract Views
875
rank
270,637
PlumX Metrics