How Hard Is It to Pick the Right Model?
14 Pages Posted: 3 Jan 2018 Last revised: 8 Jan 2018
Date Written: December 2017
Model selection has become a challenging and pressing need with recent advances in machine learning, artificial intelligence, and the availability of billions of high frequency data signals. However, a majority of model selection methods available in modern finance are subject to backtest overfitting. This is the probability that we select a financial strategy which outperforms during backtest but underperforms in practice. We evaluate the performance of the novel model confidence set (MCS) introduced in Hansen et al. (2011) in a machine learning trading strategy problem. We find that MCS is not robust to multiple testing and that it requires a very high signal-to-noise ratio to be utilizable. More generally, we raise awareness on the use of model selection methods in finance.
Keywords: Forecasting, Model confidence set, Model selection, Multiple testing
JEL Classification: G17, C52, C53
Suggested Citation: Suggested Citation