How Hard Is It to Pick the Right Model? MCS and Backtest Overfitting

27 Pages Posted: 3 Jan 2018 Last revised: 15 Jun 2018

See all articles by Diego Aparicio

Diego Aparicio

University of Navarra, IESE Business School

Marcos Lopez de Prado

Cornell University - Operations Research & Industrial Engineering; Abu Dhabi Investment Authority; True Positive Technologies

Date Written: December 2017

Abstract

Recent advances in machine learning, artificial intelligence, and the availability of billions of high frequency data signals have made model selection a challenging and pressing need. However, most of the model selection methods available in modern finance are subject to backtest overfitting. This is the probability that one will select a financial strategy that outperforms during backtest, but underperforms in practice. We evaluate the performance of the novel model confidence set (MCS) introduced in Hansen et al. (2011) in a simple machine learning trading strategy problem. We find that MCS is not robust to multiple testing and that it requires a very high signal-to-noise ratio to be utilizable. More generally, we raise awareness on the limitations of model selection in finance.

Keywords: Forecasting, Model confidence set, Model selection, Multiple testing

JEL Classification: G17, C52, C53

Suggested Citation

Aparicio, Diego and López de Prado, Marcos and López de Prado, Marcos, How Hard Is It to Pick the Right Model? MCS and Backtest Overfitting (December 2017). Available at SSRN: https://ssrn.com/abstract=3044740 or http://dx.doi.org/10.2139/ssrn.3044740

Diego Aparicio (Contact Author)

University of Navarra, IESE Business School ( email )

Carrer d'Arnus i de Gari, 3-7
Barcelona
Spain

Marcos López de Prado

Cornell University - Operations Research & Industrial Engineering ( email )

237 Rhodes Hall
Ithaca, NY 14853
United States

HOME PAGE: http://www.orie.cornell.edu

Abu Dhabi Investment Authority ( email )

211 Corniche Road
Abu Dhabi, Abu Dhabi PO Box3600
United Arab Emirates

HOME PAGE: http://www.adia.ae

True Positive Technologies ( email )

NY
United States

HOME PAGE: http://www.truepositive.com

Do you have a job opening that you would like to promote on SSRN?

Paper statistics

Downloads
935
Abstract Views
2,770
rank
35,343
PlumX Metrics