Rounding and other pitfalls in meta-studies on p-hacking and publication bias: A comment on Brodeur et al. (2020)

36 Pages Posted: 24 May 2021

Date Written: May 22, 2021

Abstract

Brodeur et al. (2020) study a large number of hypothesis tests from economic articles and find evidence for p-hacking or publication bias, in particular for studies using an IV or DID approach. We show that a crucial continuity assumption is violated because many collected estimates and standard errors are reported with just a single significant digit. These rounding errors cause bunching of computed z-statistics, particularly at exactly 2. We propose and compare different adjustments. Under every adjustment evidence for p-hacking from randomization and caliper tests at the 5% significance threshold substantially reduces. We find no more significant results for DID studies, but effects remain similar and significant for IV studies. Brodeur et al. (2020) perform additional tests that seem less affected by rounding errors. Yet, we highlight several problems of these tests.

Keywords: p-hacking, publication bias, meta-study

JEL Classification: A14, C12, C52

Suggested Citation

Kranz, Sebastian and Pütz, Peter, Rounding and other pitfalls in meta-studies on p-hacking and publication bias: A comment on Brodeur et al. (2020) (May 22, 2021). Available at SSRN: https://ssrn.com/abstract=3848786 or http://dx.doi.org/10.2139/ssrn.3848786

Sebastian Kranz (Contact Author)

University of Ulm ( email )

Albert-Einstein-Alee 11
Ulm, D-89081
Germany

HOME PAGE: http://www.uni-ulm.de/mawi/mawi-wiwi/mitarbeiter/skranz.html

Peter Pütz

Bielefeld University ( email )

Universitätsstraße 25
Bielefeld, NRW 33613
Germany

Do you have a job opening that you would like to promote on SSRN?

Paper statistics

Downloads
55
Abstract Views
232
rank
452,581
PlumX Metrics