Reusing Natural Experiments

Fisher College of Business Working Paper No. 2019-03-021

Charles A. Dice Center Working Paper No. 2019-21

69 Pages Posted: 21 Sep 2019 Last revised: 11 Aug 2021

See all articles by Davidson Heath

Davidson Heath

University of Utah - David Eccles School of Business

Matthew C. Ringgenberg

University of Utah - Department of Finance

Mehrdad Samadi

Board of Governors of the Federal Reserve System

Ingrid M. Werner

The Ohio State University - Fisher College of Business; CEPR

Date Written: August 7, 2021

Abstract

After a natural experiment is first used, other researchers often reuse the setting, examining different outcome variables. We use simulations based on real data to illustrate the multiple hypothesis testing problem that arises when researchers reuse natural experiments. We then provide guidance for future inference based on popular empirical settings including difference-in-differences regressions, instrumental variables regressions, and regression discontinuity designs. When we apply our guidance to two extensively studied natural experiments, business combination laws and the Regulation SHO pilot, we find that many results that were statistically significant using single hypothesis testing do not survive corrections for multiple hypothesis testing.

Keywords: False Positive, Identification, Multiple Hypothesis Testing, Natural Experiments

JEL Classification: G1, G10

Suggested Citation

Heath, Davidson and Ringgenberg, Matthew C. and Samadi, Mehrdad and Werner, Ingrid M., Reusing Natural Experiments (August 7, 2021). Fisher College of Business Working Paper No. 2019-03-021, Charles A. Dice Center Working Paper No. 2019-21, Available at SSRN: https://ssrn.com/abstract=3457525 or http://dx.doi.org/10.2139/ssrn.3457525

Davidson Heath

University of Utah - David Eccles School of Business ( email )

1645 E Campus Center Dr
Salt Lake City, UT 84112-9303
United States

Matthew C. Ringgenberg

University of Utah - Department of Finance ( email )

David Eccles School of Business
Salt Lake City, UT 84112
United States

Mehrdad Samadi

Board of Governors of the Federal Reserve System ( email )

20th Street and Constitution Avenue NW
Washington, DC 20551
United States

Ingrid M. Werner (Contact Author)

The Ohio State University - Fisher College of Business ( email )

2100 Neil Avenue
Columbus, OH 43210-1144
United States
614-292-6460 (Phone)
614-292-2418 (Fax)

CEPR ( email )

London
United Kingdom

Do you have a job opening that you would like to promote on SSRN?

Paper statistics

Downloads
1,226
Abstract Views
7,640
rank
20,895
PlumX Metrics