Why We Don't Really Know What ‘Statistical Significance’ Means: A Major Educational Failure

23 Pages Posted: 11 Aug 2010

See all articles by J. Scott Armstrong

J. Scott Armstrong

University of Pennsylvania - Marketing Department

Raymond Hubbard

Drake University - College of Business and Public Administration

Date Written: July 2, 2008

Abstract

The Neyman-Pearson theory of hypothesis testing, with the Type I error rate, α, as the significance level, is widely regarded as statistical testing orthodoxy. Fisher’s model of significance testing, where the evidential p value denotes the level of significance, nevertheless dominates statistical testing practice. This paradox has occurred because these two incompatible theories of classical statistical testing have been anonymously mixed together, creating the false impression of a single, coherent model of statistical inference. We show that this hybrid approach to testing, with its misleading p < α statistical significance criterion, is common in marketing research textbooks, as well as in a large random sample of papers from twelve marketing journals. That is, researchers attempt the impossible by simultaneously interpreting the p value as a Type I error rate and as a measure of evidence against the null hypothesis. The upshot is that many investigators do not know what our most cherished, and ubiquitous, research desideratum – “statistical significance” – really means. This, in turn, signals an educational failure of the first order. We suggest that tests of statistical significance, whether p’s or α’s, be downplayed in statistics and marketing research courses. Classroom instruction should focus instead on teaching students to emphasize the use of confidence intervals around point estimates in individual studies, and the criterion of overlapping confidence intervals when one has estimates from similar studies.

Suggested Citation

Armstrong, J. Scott and Hubbard, Raymond, Why We Don't Really Know What ‘Statistical Significance’ Means: A Major Educational Failure (July 2, 2008). Available at SSRN: https://ssrn.com/abstract=1154386 or http://dx.doi.org/10.2139/ssrn.1154386

J. Scott Armstrong (Contact Author)

University of Pennsylvania - Marketing Department ( email )

700 Jon M. Huntsman Hall
3730 Walnut Street
Philadelphia, PA 19104-6340
United States
215-898-5087 (Phone)
215-898-2534 (Fax)

HOME PAGE: http://marketing.wharton.upenn.edu/people/faculty/armstrong.cfm

Raymond Hubbard

Drake University - College of Business and Public Administration ( email )

2507 University Avenue
Des Moines, IA 50311-4505
United States
515-271-2344 (Phone)
515-271-4518 (Fax)

Register to save articles to
your library

Register

Paper statistics

Downloads
72
Abstract Views
585
rank
324,613
PlumX Metrics