Why We Don't Really Know What Statistical Significance Means: A Major Educational Failure
Journal of Marketing Education, Vol. 28, pp. 114-120, August 2006
23 Pages Posted: 25 May 2007 Last revised: 30 Dec 2011
The Neyman-Pearson theory of hypothesis testing, with the Type I error rate, ±, as the significance level, is widely regarded as statistical testing orthodoxy. Fisher's model of significance testing, where the evidential p value denotes the level of significance, nevertheless dominates statistical testing practice. This paradox has occurred because these two incompatible theories of classical statistical testing have been anonymously mixed together, creating the false impression of a single, coherent model of statistical inference. We show that this hybrid approach to testing, with its misleading p < ± statistical significance criterion, is common in marketing research textbooks, as well as in a large random sample of papers from twelve marketing journals. That is, researchers attempt the impossible by simultaneously interpreting the p value as a Type I error rate and as a measure of evidence against the null hypothesis. The upshot is that many investigators do not know what our most cherished, and ubiquitous, research desideratum - statistical significance - really means. This, in turn, signals an educational failure of the first order. We suggest that tests of statistical significance, whether p's or ±'s, be downplayed in statistics and marketing research courses. Classroom instruction should focus instead on teaching students to emphasize the use of confidence intervals around point estimates in individual studies, and the criterion of overlapping confidence intervals when one has estimates from similar studies.
Keywords: ± levels, p values, p < ± criterion, Fisher, Neyman-Pearson, (overlapping)
Suggested Citation: Suggested Citation
Do you have a job opening that you would like to promote on SSRN?
Are Null Results Becoming an Endangered Species in Marketing?
Replications and Extensions in Marketing - Rarely Published But Quite Contrary
Entrepreneurial Orientation and Business Performance - A Replication Study
By Hermann Frank, Alexander Kessler, ...
Why We Don't Really Know What ‘Statistical Significance’ Means: A Major Educational Failure
Cross-Cultural Comparison of Food in the Children's Media Environment in New Zealand and Japan
By Sandy Bulmer, Lynne C. Eagle, ...
The Effects of Negative Publicity on Consumer Attitudes: A Replication and Extension
Publication Bias Against Null Results
Forecasting Elections Using Expert Surveys: An Application to U.S. Presidential Elections
Editorial: Well Documented Articles Achieve More Impact
By Sönke Albers
The Impact of Gartner’s Maturity Curve, Adoption Curve, Strategic Technologies on Information Systems Research, with Applications to Artificial Intelligence, ERP, BPM and RFID