Natural Experiment Policy Evaluation: A Critique

Abstract

We argue exogenous random treatment is insufficient for valid inference regarding the sign and magnitude of causal effects in dynamic environments. In such settings, treatment responses must be understood as contingent upon the typically unmodeled policy generating process. With binary assignment, this results in quantitatively significant attenuation bias. With more than two policy states, treatment responses can be biased downward, upward, or have the wrong sign. Further, it is not only generally invalid to extrapolate elasticities across policy processes, as argued by Lucas (1976), but also to extrapolate within the same policy process. We derive auxiliary assumptions beyond exogeneity for valid inference in dynamic settings. If all possible policy transitions are rare events, treatment responses approximate causal effects. However, reliance on rare events is overly-restrictive as the necessary and sufficient conditions for equality of treatment responses and causal effects is that policy variable changes have mean zero. If these conditions are not met, we show how treatment responses can nevertheless be corrected and mapped back to causal effects or extrapolated to forecast responses to future policy changes.

Keywords: causal effects, dynamic environments, natural experiments, policy evaluations, random treatment