Rounding and other pitfalls in meta-studies on p-hacking and publication bias: A comment on Brodeur et al. (2020)
36 Pages Posted: 24 May 2021
Date Written: May 22, 2021
Brodeur et al. (2020) study a large number of hypothesis tests from economic articles and find evidence for p-hacking or publication bias, in particular for studies using an IV or DID approach. We show that a crucial continuity assumption is violated because many collected estimates and standard errors are reported with just a single significant digit. These rounding errors cause bunching of computed z-statistics, particularly at exactly 2. We propose and compare different adjustments. Under every adjustment evidence for p-hacking from randomization and caliper tests at the 5% significance threshold substantially reduces. We find no more significant results for DID studies, but effects remain similar and significant for IV studies. Brodeur et al. (2020) perform additional tests that seem less affected by rounding errors. Yet, we highlight several problems of these tests.
Keywords: p-hacking, publication bias, meta-study
JEL Classification: A14, C12, C52
Suggested Citation: Suggested Citation