How Hard Is It to Pick the Right Model?

14 Pages Posted: 3 Jan 2018 Last revised: 8 Jan 2018

Diego Aparicio

Massachusetts Institute of Technology (MIT), Department of Economics

Marcos Lopez de Prado

Lawrence Berkeley National Laboratory; True Positive Technologies; RCC - Harvard University

Date Written: December 2017

Abstract

Model selection has become a challenging and pressing need with recent advances in machine learning, artificial intelligence, and the availability of billions of high frequency data signals. However, a majority of model selection methods available in modern finance are subject to backtest overfitting. This is the probability that we select a financial strategy which outperforms during backtest but underperforms in practice. We evaluate the performance of the novel model confidence set (MCS) introduced in Hansen et al. (2011) in a machine learning trading strategy problem. We find that MCS is not robust to multiple testing and that it requires a very high signal-to-noise ratio to be utilizable. More generally, we raise awareness on the use of model selection methods in finance.

Keywords: Forecasting, Model confidence set, Model selection, Multiple testing

JEL Classification: G17, C52, C53

Suggested Citation

Aparicio, Diego and Lopez de Prado, Marcos, How Hard Is It to Pick the Right Model? (December 2017). Available at SSRN: https://ssrn.com/abstract=3044740 or http://dx.doi.org/10.2139/ssrn.3044740

Diego Aparicio (Contact Author)

Massachusetts Institute of Technology (MIT), Department of Economics ( email )

Cambridge, MA
United States

Marcos Lopez de Prado

Lawrence Berkeley National Laboratory ( email )

1 Cyclotron Road
Berkeley, CA 94720
United States

HOME PAGE: http://www.lbl.gov

True Positive Technologies ( email )

12 East 49th Street, Floor 37
New York, NY 10017
United States

HOME PAGE: http://www.truepositive.com

RCC - Harvard University ( email )

1875 Cambridge Street
Cambridge, MA 02138
United States

HOME PAGE: http://www.rcc.harvard.edu

Register to save articles to
your library

Register

Paper statistics

Downloads
264
rank
104,545
Abstract Views
705
PlumX