Do Pre-Registration and Pre-Analysis Plans Reduce P-Hacking and Publication Bias?

Randomized controlled trials (RCTs) are increasingly prominent in economics, with pre-registration and pre-analysis plans (PAPs) promoted as important in ensuring the credibility of findings. We investigate whether these tools reduce the extent of p-hacking and publication bias by collecting and studying the universe of test statistics, 15,992 in total, from RCTs published in 15 leading economics journals from 2018 through 2021. In our primary analysis, we find no meaningful difference in the distribution of test statistics from pre-registered studies, compared to their non-pre-registered counterparts. However, pre-registerd studies that have a complete PAP are significantly less p-hacked. This results point to the importance of PAPs, rather than pre-registration in itself, in ensuring credibility.

Keywords: research credibility, publication bias, p-hacking, pre-registration, pre-analysis plan

JEL Classification: B41, C13, C40, C93

Suggested Citation: Suggested Citation

Brodeur, Abel and Brodeur, Abel and Cook, Nikolai and Hartley, Jonathan and Heyes, Anthony, Do Pre-Registration and Pre-Analysis Plans Reduce P-Hacking and Publication Bias?. IZA Discussion Paper No. 15476, Available at SSRN: https://ssrn.com/abstract=4188287