P-Hacking in Experimental Accounting Studies

Abstract

We study the extent and determinants of p-hacking in experimental accounting studies. Based on p-values in experimental studies published in top accounting journals, we find an unusual abundance of p-values that are just significant: the observed frequency of p-values equal to 0.05 is 22% to 30% higher than what would be expected based on the frequency of other p-values. This suggests that some researchers may have p-hacked to obtain “just significant” results. Further analyses reveal that p-hacking is more evident in articles by junior researchers, authors from highly ranked schools, male authors, or sole authors. Moreover, the p-hacking evidence is stronger in articles with more experiment participants or in tests concerning key results. More cited articles exhibit a greater extent of p-hacking, indicating that p-hacking helps attract citations. Lastly, we provide the first evidence that the disclosure policy adopted by the Journal of Accounting Research in 2014 mitigates p-hacking.

Keywords: P-hacking; Experimental accounting; NHST; Statistical significance; Hypothesis testing