Rounding and other pitfalls in meta-studies on p-hacking and publication bias: A comment on Brodeur et al. (2020)

Abstract

Brodeur et al. (2020) study a large number of hypothesis tests from economic articles and find evidence for p-hacking or publication bias, in particular for studies using an IV or DID approach. We show that a crucial continuity assumption is violated because many collected estimates and standard errors are reported with just a single significant digit. These rounding errors cause bunching of computed z-statistics, particularly at exactly 2. We propose and compare different adjustments. Under every adjustment evidence for p-hacking from randomization and caliper tests at the 5% significance threshold substantially reduces. We find no more significant results for DID studies, but effects remain similar and significant for IV studies. Brodeur et al. (2020) perform additional tests that seem less affected by rounding errors. Yet, we highlight several problems of these tests.

Keywords: p-hacking, publication bias, meta-study

JEL Classification: A14, C12, C52