Natural Experiment Policy Evaluation: A Critique

54 Pages Posted: 2 Mar 2015

See all articles by Chris Hennessy

Chris Hennessy

London Business School

Ilya A. Strebulaev

Stanford University - Graduate School of Business; National Bureau of Economic Research

Multiple version iconThere are 3 versions of this paper

Date Written: March 2015

Abstract

We argue exogenous random treatment is insufficient for valid inference regarding the sign and magnitude of causal effects in dynamic environments. In such settings, treatment responses must be understood as contingent upon the typically unmodeled policy generating process. With binary assignment, this results in quantitatively significant attenuation bias. With more than two policy states, treatment responses can be biased downward, upward, or have the wrong sign. Further, it is not only generally invalid to extrapolate elasticities across policy processes, as argued by Lucas (1976), but also to extrapolate within the same policy process. We derive auxiliary assumptions beyond exogeneity for valid inference in dynamic settings. If all possible policy transitions are rare events, treatment responses approximate causal effects. However, reliance on rare events is overly-restrictive as the necessary and sufficient conditions for equality of treatment responses and causal effects is that policy variable changes have mean zero. If these conditions are not met, we show how treatment responses can nevertheless be corrected and mapped back to causal effects or extrapolated to forecast responses to future policy changes.

Keywords: causal effects, dynamic environments, natural experiments, policy evaluations, random treatment

Suggested Citation

Hennessy, Christopher and Strebulaev, Ilya A., Natural Experiment Policy Evaluation: A Critique (March 2015). CEPR Discussion Paper No. DP10455. Available at SSRN: https://ssrn.com/abstract=2572453

Christopher Hennessy (Contact Author)

London Business School ( email )

Sussex Place
Regent's Park
London, London NW1 4SA
United Kingdom

Ilya A. Strebulaev

Stanford University - Graduate School of Business ( email )

655 Knight Way
Stanford, CA 94305-5015
United States

HOME PAGE: http://faculty-gsb.stanford.edu/strebulaev/

National Bureau of Economic Research ( email )

1050 Massachusetts Avenue
Cambridge, MA 02138
United States

Here is the Coronavirus
related research on SSRN

Paper statistics

Downloads
1
Abstract Views
498
PlumX Metrics