How Hard Is It to Pick the Right Model? MCS and Backtest Overfitting
27 Pages Posted: 3 Jan 2018 Last revised: 15 Jun 2018
Date Written: December 2017
Recent advances in machine learning, artificial intelligence, and the availability of billions of high frequency data signals have made model selection a challenging and pressing need. However, most of the model selection methods available in modern finance are subject to backtest overfitting. This is the probability that one will select a financial strategy that outperforms during backtest, but underperforms in practice. We evaluate the performance of the novel model confidence set (MCS) introduced in Hansen et al. (2011) in a simple machine learning trading strategy problem. We find that MCS is not robust to multiple testing and that it requires a very high signal-to-noise ratio to
be utilizable. More generally, we raise awareness on the limitations of model selection in finance.
Keywords: Forecasting, Model confidence set, Model selection, Multiple testing
JEL Classification: G17, C52, C53
Suggested Citation: Suggested Citation