How Much Should We Trust Instrumental Variable Estimates in Political Science? Practical Advice based on Over 60 Replicated Studies
260 Pages Posted: 16 Aug 2021 Last revised: 31 Mar 2023
Date Written: March 30, 2023
Instrumental variable (IV) strategies are widely used in political science to establish causal relationships, but the identifying assumptions required by an IV design are demanding, and assessing their validity remains challenging. In this paper, we replicate 67 papers published in three top political science journals from 2010-2022 and identify several concerning patterns. First, researchers often overestimate the strength of their instruments due to non-i.i.d. error structures such as clustering. Second, the commonly used $t$-test for two-stage-least-squares (2SLS) estimates frequently underestimates uncertainty. Using more robust inferential methods, we find that about 19-30\% of the 2SLS estimates in our sample are underpowered. Third, in most replicated studies, 2SLS estimates are significantly larger than ordinary-least-squares estimates, with their ratio negatively correlated with instrument strength in studies with non-experimentally generated instruments, suggesting potential violations of unconfoundedness or exclusion restriction. We provide a checklist and software to help researchers avoid these pitfalls and improve their practice.
Keywords: instrumental variables, two-stage-least-squared, replications, weak instrument, exclusion restriction, replication
JEL Classification: C26
Suggested Citation: Suggested Citation