How Much Should We Trust Instrumental Variable Estimates in Political Science? Practical Advice based on Over 60 Replicated Studies

Abstract

Instrumental variable (IV) strategies are commonly used in political science to establish causal relationships, yet the identifying assumptions required by an IV design are demanding and it remains challenging for researchers to evaluate their plausibility. We replicate 61 papers published in three top journals in political science from the past decade (2010-2020) and document several troubling patterns: (1) researchers often miscalculate the first-stage F statistics, overestimating the strength of their IVs; (2) most researchers rely on classical asymptotic standard errors, which often severely underestimate the uncertainties around the two-stage-least-squares (2SLS) estimates; (3) in the majority of the replicated studies, the 2SLS estimates are much bigger than the ordinary-least-squares estimates, and their ratio is negatively correlated with the strength of the IVs in studies where the IVs are not experimentally generated, suggesting potential violations of the exclusion restriction. To improve practice, we provide a checklist for researchers to avoid these pitfalls and recommend a zero-first-stage test and a local-to-zero procedure to guard against failures of the identifying assumptions.

Keywords: instrumental variables, two-stage-least-squared, replications, weak IV, exclusion restriction, zero-first-stage test

JEL Classification: C26