The Trouble with Instruments: The Need for Pre-Treatment Balance in Shock-IV Designs

Abstract

Credible causal inference in accounting and finance research often comes from “natural” experiments. These experiments can be exploited using several “shock-based” research designs, including difference-in-differences (DiD), instrumental variables based on the shock (shock-IV), and regression discontinuity (RD). We study here shock-IV designs using panel data. Shock-IVs are potentially more credible than non-shock IVs, but can be problematic unless carefully handled. We identify all shock-IV papers in two broad datasets; and re-examine three of the apparently strongest papers – Duchin, Matsusaka and Ozbas (“DMO,” JFE 2010); Iliev (JF 2010); and Desai and Dharmapala (REStat 2009). We show that the IVs in all three papers are unusably weak – once we enforce covariate balance and common support for treated and control firms, the instruments are no longer significant in the first stage. All three papers also show non-parallel pre-treatment trends on outcomes or core covariates. The problems with these papers generalize to our full sample, and to other papers exploiting the same shocks as DMO. A core conclusion of our reexamination is that “pre-treatment balance” (common support, covariate balance, and parallel pre-treatment trends) is necessary for credible shock-IV designs. We provide a good practice checklist for shock-IV design with panel data. Finding valid, usable shock-IVs is more challenging than prior research suggests.

The Online Appendix is available at: http://ssrn.com/abstract=2859113.

Keywords: Instrumental variables; shock-based research design; exclusion restriction; covariate balance

JEL Classification: C26, G34, G38