P-Hacking in Experimental Accounting Studies

55 Pages Posted: 12 Jan 2021 Last revised: 10 Mar 2021

See all articles by Xin (Simba) Chang

Xin (Simba) Chang

Nanyang Business School, Nanyang Technological University

Huasheng Gao

Fanhai International School of Finance, Fudan University

Wei Li

Shanghai University of Finance and Economics

Date Written: January 8, 2021

Abstract

We study the extent and determinants of p-hacking in experimental accounting studies. Using a text-mining approach to extract p-values from experimental studies published in three highly regarded accounting journals, we find an unusual abundance of p-values that are just significant: the observed frequency of p-values equal to 0.05 is about 26% higher than what would be expected based on the frequency of other p-values. This suggests that some researchers may have p-hacked to obtain “just significant” results. Further analyses reveal that p-hacking is more evident in articles by untenured authors, early-career researchers, authors from highly ranked schools, male authors, or sole authors. Moreover, the p-hacking evidence is stronger in articles with more experiment participants or in tests concerning key results. More cited articles exhibit a greater extent of p-hacking, indicating that p-hacking helps attract citations. Lastly, we document that the data and code disclosure policy adopted by the Journal of Accounting Research in 2014 mitigates p-hacking.

Keywords: P-hacking; Experimental accounting; NHST; Statistical significance; Hypothesis testing

Suggested Citation

Chang, Xin and Gao, Huasheng and Li, Wei, P-Hacking in Experimental Accounting Studies (January 8, 2021). Nanyang Business School Research Paper No. 21-01, Available at SSRN: https://ssrn.com/abstract=3762342 or http://dx.doi.org/10.2139/ssrn.3762342

Xin Chang (Contact Author)

Nanyang Business School, Nanyang Technological University ( email )

S3-01B-40 Nanyang Avenue
Singapore, 639798
Singapore

HOME PAGE: https://personal.ntu.edu.sg/changxin/

Huasheng Gao

Fanhai International School of Finance, Fudan University ( email )

Beijing West District Baiyun Load 10th
Shanghai, 100045
China
2165642222 (Phone)
2165642222 (Fax)

Wei Li

Shanghai University of Finance and Economics

Guoding Rd 777
Yangpu District
Shanghai, 200433
China
(86) 021-65908840 (Phone)

Do you have a job opening that you would like to promote on SSRN?

Paper statistics

Downloads
199
Abstract Views
456
rank
181,365
PlumX Metrics