Trusting Difference-in-Differences Estimates More: An Approximate Permutation Test

Abstract

Researchers use difference-in-differences models to evaluate the causal effects of policy changes. As the empirical correlation across firms and time can be ambiguous, estimating consistent standard errors is difficult and statistical inferences may be biased. We apply an approximate permutation test using simulated interventions to reveal the empirical error distribution of estimated policy effects. In contrast to existing econometric corrections, such as single- or double-clustering, this approach does not impose any parametric form on the data. In comparison with alternative parametric tests, this procedure maintains correct size with simulated and real-world interventions. Simultaneously, it improves power.

Keywords: difference-in-differences, policy studies, clustered standard errors, placebo laws, approximate permutation tests

JEL Classification: C12, C14, C93