False (and Missed) Discoveries in Financial Economics
89 Pages Posted: 21 Nov 2017 Last revised: 15 Aug 2019
Date Written: August 14, 2019
The multiple testing problem plagues many important issues in finance such as fund and factor selection. Many look good purely by luck. There are a number of statistical techniques to control for multiplicity that reduce Type I errors - but it is unknown by how much. We propose a new way to calibrate both Type I and Type II errors. We start with the researcher's prior belief on the proportion of managers that are skilled. Using a double bootstrap method, we then establish a t-statistic hurdle that is associated with a specific false discovery rate (e.g., 5%). We also establish a t-statistic hurdle that is associated with a certain acceptable ratio of misses to false discoveries (Type II error scaled by Type I error) | effectively allowing for differential costs of the two types of mistakes. Evaluating current methods within our framework, we find that they lack the power to detect outperforming managers.
Keywords: Type I, Type II, Multiple testing, False discoveries, Odds ratio, Power, Mutual funds, Smart beta, Anomalies, Bayesian, Factors, Backtesting, Factor Zoo
JEL Classification: G12, G14, C12, C21, C22, C31, C32
Suggested Citation: Suggested Citation