P-Hacking in Experimental Accounting Studies

Abstract

We study the extent and determinants of p-hacking in experimental accounting studies. Using a text-mining approach to extract p-values from experimental studies published in three highly regarded accounting journals, we find an unusual abundance of p-values that are just significant: the observed frequency of p-values equal to 0.05 is about 26% higher than what would be expected based on the frequency of other p-values. This suggests that some researchers may have p-hacked to obtain “just significant” results. Further analyses reveal that p-hacking is more evident in articles by untenured authors, early-career researchers, authors from highly ranked schools, male authors, or sole authors. Moreover, the p-hacking evidence is stronger in articles with more experiment participants or in tests concerning key results. More cited articles exhibit a greater extent of p-hacking, indicating that p-hacking helps attract citations. Lastly, we document that the data and code disclosure policy adopted by the Journal of Accounting Research in 2014 mitigates p-hacking.

Keywords: P-hacking; Experimental accounting; NHST; Statistical significance; Hypothesis testing