What to Look for in a Backtest

58 Pages Posted: 12 Aug 2013 Last revised: 5 Jul 2015

See all articles by Marcos Lopez de Prado

Marcos Lopez de Prado

Cornell University - Operations Research & Industrial Engineering; Abu Dhabi Investment Authority; True Positive Technologies

Date Written: August 11, 2013

Abstract

A large number of quantitative hedge funds have historically sustained losses. In this study we argue that the backtesting methodology at the core of their strategy selection process may have played a role.

* Most firms and portfolio managers rely on backtests (or historical simulations of performance) to allocate capital to investment strategies.

* After trying only 7 strategy configurations, a researcher is expected to identify at least one 2-year long backtest with an annualized Sharpe ratio of over 1, when the expected out of sample Sharpe ratio is 0.

* If the researcher tries a large enough number of strategy configurations, a backtest can always be fit to any desired performance for a fixed sample length. Thus, there is a minimum backtest length (MinBTL) that should be required for a given number of trials.

* Standard statistical techniques designed to prevent regression over-fitting, such as hold-out, are inaccurate in the context of backtest evaluation.

* The practical totality of published backtests do not report the number of trials involved.

* Under memory effects, over-fitting leads to systematic losses, not noise.

Keywords: backtest, historical simulation, probability of backtest over-fitting, investment strategy, optimization, Sharpe ratio, minimum back-test length, performance degradation

JEL Classification: G0, G1, G2, G15, G24, E44

Suggested Citation

López de Prado, Marcos and López de Prado, Marcos, What to Look for in a Backtest (August 11, 2013). Available at SSRN: https://ssrn.com/abstract=2308682 or http://dx.doi.org/10.2139/ssrn.2308682

Marcos López de Prado (Contact Author)

Cornell University - Operations Research & Industrial Engineering ( email )

237 Rhodes Hall
Ithaca, NY 14853
United States

HOME PAGE: http://www.orie.cornell.edu

Abu Dhabi Investment Authority ( email )

211 Corniche Road
Abu Dhabi, Abu Dhabi PO Box3600
United Arab Emirates

HOME PAGE: http://www.adia.ae

True Positive Technologies ( email )

NY
United States

HOME PAGE: http://www.truepositive.com

Do you have negative results from your research you’d like to share?

Paper statistics

Downloads
6,062
Abstract Views
25,871
Rank
2,408
PlumX Metrics