The Probability of Backtest Overfitting

Journal of Computational Finance (Risk Journals), 2015, Forthcoming

34 Pages Posted: 16 Sep 2013 Last revised: 5 Jul 2015

See all articles by David H. Bailey

David H. Bailey

Lawrence Berkeley National Laboratory; University of California, Davis

Jonathan Borwein

University of Newcastle (Australia); Royal Society of Canada; Australian Academy of Science (deceased)

Marcos Lopez de Prado

Cornell University - Operations Research & Industrial Engineering; AQR Capital Management, LLC

Qiji Jim Zhu

Western Michigan University

Multiple version iconThere are 2 versions of this paper

Date Written: February 27, 2015

Abstract

Most firms and portfolio managers rely on backtests (or historical simulations of performance) to select investment strategies and allocate them capital. Standard statistical techniques designed to prevent regression over-fitting, such as hold-out, tend to be unreliable and inaccurate in the context of investment backtests. We propose a framework that estimates the probability of backtest over-fitting (PBO) specifically in the context of investment simulations, through a numerical method that we call combinatorially symmetric cross-validation (CSCV). We show that CSCV produces accurate estimates of the probability that a particular backtest is over-fit.

The appendices for this paper are available at the following URL: http://ssrn.com/abstract=2568435

Keywords: backtest, historical simulation, probability of backtest over-fitting, investment strategy, optimization, Sharpe ratio, minimum backtest length, performance degradation

JEL Classification: G0, G1, G2, G15, G24, E44

Suggested Citation

Bailey, David H. and Borwein, Jonathan and López de Prado, Marcos and Zhu, Qiji Jim, The Probability of Backtest Overfitting (February 27, 2015). Journal of Computational Finance (Risk Journals), 2015, Forthcoming. Available at SSRN: https://ssrn.com/abstract=2326253 or http://dx.doi.org/10.2139/ssrn.2326253

David H. Bailey

Lawrence Berkeley National Laboratory ( email )

1 Cyclotron Road
Berkeley, CA 94720
United States

HOME PAGE: http://www.davidhbailey.com

University of California, Davis ( email )

One Shields Avenue
Davis, CA 95616
United States

HOME PAGE: http://www.davidhbailey.com

Jonathan Borwein

University of Newcastle (Australia)

Royal Society of Canada

Australian Academy of Science (deceased)

Marcos López de Prado (Contact Author)

Cornell University - Operations Research & Industrial Engineering ( email )

237 Rhodes Hall
Ithaca, NY 14853
United States

HOME PAGE: http://www.orie.cornell.edu

AQR Capital Management, LLC

One Greenwich Plaza
Greenwich, CT 06830
United States

HOME PAGE: http://www.aqr.com

Qiji Jim Zhu

Western Michigan University ( email )

Kalamazoo, MI 49008
United States

Register to save articles to
your library

Register

Paper statistics

Downloads
9,894
rank
438
Abstract Views
35,498
PlumX Metrics
!

Under construction: SSRN citations while be offline until July when we will launch a brand new and improved citations service, check here for more details.

For more information