'Statistics Gone on Holiday': Misinterpretations of Hypothesis Tests Propagated by Internet Resources

Abstract

“Type I error” is a basic concept in statistical hypothesis testing. However, the term is used in two subtly different senses in statistics texts and other statistical literature. Specifically, type I error can be construed either as a conditional event (i.e. presuming that the null hypothesis is true) or an unconditional event. We explain the distinctions between the different usages of type I error, and we conduct a logical analysis of popular statistics web sites to determine their usage of the terminology. Our analysis shows that ambiguous and inconsistent usage of this terminology leads to wrong interpretations of significance level in many web pages, leading in turn to faulty interpretations of the results of experiments. We discuss the reasons for this long-standing lack of consensus in the definition of type I error. The unconditional-event definition is more intuitive and agrees with the original formulation Neyman and Pearson in 1933, but professional statisticians favor the conditional-event definition. The fact that users of statistics come from widely different fields makes it difficult to arrive at a single agreed-upon definition. We conclude that even in a rigorous technical subject like statistics, ambiguous terminology can go unrecognized and can continue to produce errors in reasoning.

Keywords: Conditional probability, hypothesis testing, significance level, Type I error, web pages, internet, statistical concepts

JEL Classification: C12, C18