Rounding and Other Pitfalls in Meta-Studies on P-Hacking and Publication Bias: A Comment on Brodeur Et Al. (2020)

Abstract

Brodeur et al. (2020) study hypothesis tests from economic articles and find evidence for p-hacking and publication bias, in particular for IV and DID studies. We adjust for rounding errors in reported estimates and standard errors using different approaches. Statistical evidence for p-hacking from randomization tests and caliper tests at the 5% significance threshold robustly vanishes for DID studies but remains for IV studies. In addition, BCH derive latent distribution of z-statistics absent publication bias using two different approaches. We show that neither approach is applicable due to reasons unrelated to rounding.

Keywords: p-hacking, publication bias, meta-study

JEL Classification: A14, C12, C52