Download this Paper Open PDF in Browser

The Probability of Backtest Overfitting

Journal of Computational Finance (Risk Journals), 2015, Forthcoming

34 Pages Posted: 16 Sep 2013 Last revised: 5 Jul 2015

David H. Bailey

Lawrence Berkeley National Laboratory; University of California, Davis

Jonathan M. Borwein

University of Newcastle (Australia); Royal Society of Canada; Australian Academy of Science

Marcos Lopez de Prado

Guggenheim Partners, LLC; Lawrence Berkeley National Laboratory; Harvard University - RCC

Qiji Jim Zhu

Western Michigan University

Multiple version iconThere are 2 versions of this paper

Date Written: February 27, 2015

Abstract

Most firms and portfolio managers rely on backtests (or historical simulations of performance) to select investment strategies and allocate them capital. Standard statistical techniques designed to prevent regression over-fitting, such as hold-out, tend to be unreliable and inaccurate in the context of investment backtests. We propose a framework that estimates the probability of backtest over-fitting (PBO) specifically in the context of investment simulations, through a numerical method that we call combinatorially symmetric cross-validation (CSCV). We show that CSCV produces accurate estimates of the probability that a particular backtest is over-fit.

The appendices for this paper are available at the following URL: http://ssrn.com/abstract=2568435

Keywords: backtest, historical simulation, probability of backtest over-fitting, investment strategy, optimization, Sharpe ratio, minimum backtest length, performance degradation

JEL Classification: G0, G1, G2, G15, G24, E44

Suggested Citation

Bailey, David H. and Borwein, Jonathan M. and Lopez de Prado, Marcos and Zhu, Qiji Jim, The Probability of Backtest Overfitting (February 27, 2015). Journal of Computational Finance (Risk Journals), 2015, Forthcoming. Available at SSRN: https://ssrn.com/abstract=2326253 or http://dx.doi.org/10.2139/ssrn.2326253

David H. Bailey

Lawrence Berkeley National Laboratory ( email )

1 Cyclotron Road
Berkeley, CA 94720
United States

HOME PAGE: http://www.davidhbailey.com

University of California, Davis ( email )

One Shields Avenue
Davis, CA 95616
United States

HOME PAGE: http://www.davidhbailey.com

Jonathan M. Borwein

University of Newcastle (Australia) ( email )

University Drive
Callaghan, NSW 2308
Australia
+61 2 4921 5535 (Phone)

HOME PAGE: http://www.carma.newcastle.edu.au/jon

Royal Society of Canada ( email )

282 Somerset West
Ottawa, Ontario K2P 0J6
Canada

Australian Academy of Science ( email )

Ian Potter House
Gordon Street
Acton, ACT 2601
Australia

Marcos Lopez de Prado (Contact Author)

Guggenheim Partners, LLC ( email )

330 Madison Avenue
New York, NY 10017
United States

HOME PAGE: http://www.QuantResearch.org

Lawrence Berkeley National Laboratory ( email )

1 Cyclotron Road
Berkeley, CA 94720
United States

HOME PAGE: http://www.lbl.gov

Harvard University - RCC ( email )

26 Trowbridge Street
Cambridge, MA 02138
United States

HOME PAGE: http://www.rcc.harvard.edu

Qiji Jim Zhu

Western Michigan University ( email )

Kalamazoo, MI 49008
United States

Paper statistics

Downloads
7,239
Rank
598
Abstract Views
24,630