How Much Should We Trust Instrumental Variable Estimates in Political Science? Practical Advice based on Over 60 Replicated Studies

Abstract

Instrumental variable (IV) strategies are widely used in political science to establish causal relationships, but the identifying assumptions required by an IV design are demanding, and assessing their validity remains challenging. In this paper, we replicate 67 papers published in three top political science journals from 2010-2022 and identify several concerning patterns. First, researchers often overestimate the strength of their instruments due to non-i.i.d. error structures such as clustering. Second, the commonly used $t$-test for two-stage-least-squares (2SLS) estimates frequently underestimates uncertainty. Using more robust inferential methods, we find that about 19-30\% of the 2SLS estimates in our sample are underpowered. Third, in most replicated studies, 2SLS estimates are significantly larger than ordinary-least-squares estimates, with their ratio negatively correlated with instrument strength in studies with non-experimentally generated instruments, suggesting potential violations of unconfoundedness or exclusion restriction. We provide a checklist and software to help researchers avoid these pitfalls and improve their practice.

Keywords: instrumental variables, two-stage-least-squared, replications, weak instrument, exclusion restriction, replication

JEL Classification: C26