False (and Missed) Discoveries in Financial Economics
60 Pages Posted: 21 Nov 2017 Last revised: 2 Aug 2018
Date Written: July 25, 2018
Investors make two types of mistakes. First, they erroneously allocate to an asset manager (or a “smart” beta) that underperforms because the asset manager lacks skill. Second, investors might miss out allocating to a good manager. The first mistake is difficult to deal with given there are thousands of managers and many look good purely by luck. We introduce a new technique that optimizes the threshold for a prespecified false discovery rate (i.e., chance of the first mistake), at say 5%. Our method also allows for heterogeneous false discoveries – we should not treat all bad managers the same because some are really, really bad. Next, we focus on the second type of error where investors miss out on good managers. It is routine to ignore this type of mistake. Our results show that current research methods have little or no power to detect good managers. Finally, our method allows for the asymmetric treatment of false discoveries and misses – generally, investing in a bad manager is more costly than missing a good manager. We also offer a way to select managers whereby the investor can prespecify the ratio of false discoveries to misses to accommodate these differential costs. For instance, we can accommodate a decision rule whereby the investor is willing to miss ten good managers to avoid the mistake of selecting one bad manager.
Keywords: Type I, Type II, Multiple testing, False discoveries, Odds ratio, Power, Mutual funds, Smart beta, Anomalies, Bayesian, Factors, Backtesting, Factor Zoo
JEL Classification: G12, G14, C12, C21, C22, C31, C32
Suggested Citation: Suggested Citation