How Hard Is It to Pick the Right Model? MCS and Backtest Overfitting

27 Pages Posted: 3 Jan 2018 Last revised: 15 Jun 2018

Diego Aparicio

Massachusetts Institute of Technology (MIT), Department of Economics

Marcos Lopez de Prado

Cornell University - Operations Research & Industrial Engineering; Lawrence Berkeley National Laboratory; RCC - Harvard University

Date Written: December 2017

Abstract

Recent advances in machine learning, artificial intelligence, and the availability of billions of high frequency data signals have made model selection a challenging and pressing need. However, most of the model selection methods available in modern finance are subject to backtest overfitting. This is the probability that one will select a financial strategy that outperforms during backtest, but underperforms in practice. We evaluate the performance of the novel model confidence set (MCS) introduced in Hansen et al. (2011) in a simple machine learning trading strategy problem. We find that MCS is not robust to multiple testing and that it requires a very high signal-to-noise ratio to
be utilizable. More generally, we raise awareness on the limitations of model selection in finance.

Keywords: Forecasting, Model confidence set, Model selection, Multiple testing

JEL Classification: G17, C52, C53

Suggested Citation

Aparicio, Diego and Lopez de Prado, Marcos, How Hard Is It to Pick the Right Model? MCS and Backtest Overfitting (December 2017). Available at SSRN: https://ssrn.com/abstract=3044740 or http://dx.doi.org/10.2139/ssrn.3044740

Diego Aparicio (Contact Author)

Massachusetts Institute of Technology (MIT), Department of Economics ( email )

Cambridge, MA
United States

Marcos López de Prado

Cornell University - Operations Research & Industrial Engineering ( email )

237 Rhodes Hall
Ithaca, NY 14853
United States

HOME PAGE: http://www.orie.cornell.edu

Lawrence Berkeley National Laboratory ( email )

1 Cyclotron Road
Berkeley, CA 94720
United States

HOME PAGE: http://www.lbl.gov

RCC - Harvard University ( email )

1875 Cambridge Street
Cambridge, MA 02138
United States

HOME PAGE: http://www.rcc.harvard.edu

Register to save articles to
your library

Register

Paper statistics

Downloads
316
rank
86,497
Abstract Views
916
PlumX