False (and Missed) Discoveries in Financial Economics

Abstract

The multiple testing problem plagues many important issues in finance such as fund and factor selection. Many look good purely by luck. There are a number of statistical techniques to control for multiplicity that reduce Type I errors - but it is unknown by how much. We propose a new way to calibrate both Type I and Type II errors. We start with the researcher's prior belief on the proportion of managers that are skilled. Using a double bootstrap method, we then establish a t-statistic hurdle that is associated with a specific false discovery rate (e.g., 5%). We also establish a t-statistic hurdle that is associated with a certain acceptable ratio of misses to false discoveries (Type II error scaled by Type I error) | effectively allowing for differential costs of the two types of mistakes. Evaluating current methods within our framework, we find that they lack the power to detect outperforming managers.

Keywords: Type I, Type II, Multiple testing, False discoveries, Odds ratio, Power, Mutual funds, Smart beta, Anomalies, Bayesian, Factors, Backtesting, Factor Zoo

JEL Classification: G12, G14, C12, C21, C22, C31, C32