Specification Choice in Randomized and Natural Experiments: Lessons from the Regulation SHO Experiment

104 Pages Posted: 21 Jul 2020 Last revised: 13 Oct 2020

Date Written: October 11, 2020

Abstract

During 2005-2007, the SEC conducted a randomized trial in which it removed short-sale restrictions from one-third of the Russell 3000 firms. Early studies found modest microstructure-related effects of removing the restrictions but no effect on short interest or share prices. More recently, however, many studies have attributed a wide range of substantive indirect outcomes to this experiment. We revisit the principal findings in four recent studies in major journals, using a sample that closely matches the actual experiment and a pre-specified research design, and find no support for any of the reported results. If we instead match their specifications as best we can based on the published descriptions, we still obtain quite different results; only two of 13 outcomes are statistically significant, barely so, and even those results are tenuous. For two papers, we have the authors’ original data and code. We can technically replicate two (different) results, but those results are highly sensitive to specification. Our findings have implications for the robustness of other studies finding indirect effects of the short-sale experiment. More broadly, they have implications for the credibility of “causal” research designs which rely on randomization or on natural experiments. Researchers retain extensive discretion over both the sample and model specification, even for a true randomized experiment. The choices in these studies produced significant results, when other reasonable choices would not.

The Online Appendix is available at https://ssrn.com/abstract=3657200.

Our pre-specified analysis plan is available at https://ssrn.com/abstract=3415529.

Suggested Citation

Black, Bernard S. and Desai, Hemang and Litvak, Kate and Yoo, Woongsun and Yu, Jeff Jiewei, Specification Choice in Randomized and Natural Experiments: Lessons from the Regulation SHO Experiment (October 11, 2020). Northwestern Law & Econ Research Paper 20-06, Available at SSRN: https://ssrn.com/abstract=3657196 or http://dx.doi.org/10.2139/ssrn.3657196

Bernard S. Black (Contact Author)

Northwestern University - Pritzker School of Law ( email )

375 E. Chicago Ave
Chicago, IL 60611
United States
312-503-2784 (Phone)

Northwestern University - Kellogg School of Management

2001 Sheridan Road
Evanston, IL 60208
United States
847-491-5049 (Phone)

European Corporate Governance Institute (ECGI)

Brussels
Belgium

Hemang Desai

Southern Methodist University (SMU) - Accounting Department ( email )

United States
214-768-3185 (Phone)
214-768-4099 (Fax)

Kate Litvak

Northwestern University - Pritzker School of Law ( email )

375 E. Chicago Ave
Chicago, IL 60611
United States

Woongsun Yoo

Central Michigan University ( email )

Mount Pleasant, MI 48859
United States

Jeff Jiewei Yu

University of Arizona ( email )

Department of History
Tucson, AZ 85721
United States
520-621-1273 (Phone)

HOME PAGE: http://https://accounting.eller.arizona.edu/people/jeff-jiewei-yu

Here is the Coronavirus
related research on SSRN

Paper statistics

Downloads
318
Abstract Views
2,748
rank
107,689
PlumX Metrics