Significance Testing in Accounting Research: A Critical Evaluation Based on Evidence

Abstract

From a survey of the papers published in leading accounting journals in 2014, we find that accounting researchers conduct significance testing almost exclusively at a conventional level of significance, without considering the key factors such as sample size or power of a test. We present evidence that a vast majority of the accounting studies favour large or massive sample sizes and conduct significance tests with the power extremely close to or equal to one. As a result, statistical inference is severely biased towards Type I error, frequently rejecting the true null hypotheses. Under the ‘p-value less than 0.05’ criterion for statistical significance, more than 90% of the surveyed papers report statistical significance. However, under alternative criteria, only 40% of the results are statistically significant. We propose that substantial changes be made to the current practice of significance testing for more credible empirical research in accounting.

Keywords: Bayesian inference, Research credibility, Sample size, Statistical significance, Statistical power

JEL Classification: C12, M40