An Evaluation of Alternative Multiple Testing Methods for Finance Applications

Abstract

In almost every area of empirical finance, researchers are confronted with multiple tests. One high profile example is the identification of investment managers that outperform. Many beat their benchmarks purely by luck. Multiple testing methods are designed to control for luck. Factor selection is another glaring case. However, there are numerous other applications that do not get as much attention. Importantly, for example, in a simple regression model where, say, five variables are tested, a t-statistic of 2.0 is not enough to establish significance — because five variables were tried. Our paper provides a guide to various multiple testing methods and details a number of applications. We provide simulation evidence on the relative performance of different methods across a variety of testing environments. The goal of our paper is to provide a menu that researchers can choose from to improve inference in financial economics.

Keywords: Multiple hypothesis testing, False rejections, False discovery rate, False non-discovery rate, False omission rate, Family-wise error rate, Data mining, Data snooping, Type I error, Type II error, False discovery control, Luck, Test power

JEL Classification: G0, G1, G2, G3, G5, M4, C1, C5