Practical Significance of Statistical Significance

Abstract

Statistical significance is a core concept in many sciences to distinguish between what is and what is not. But what are the practical implications of finding a significant effect, and is it more reliable when based on more observations or higher confidence levels? We consider forecast accuracy in a practical setting using significant monthly deviations from the mean in a more than 300 year long time series of stock market returns. While statistical significance in general implies a marginal improved probability of correct forecasts, results vary strongly depending on significance levels and number of observations used. Surprisingly, signals based on higher significance levels or more observations do not necessarily imply better forecasts.

Keywords: Return predictability, January effect, forecasting

JEL Classification: G10