What to Look for in a Backtest
Marcos Lopez de Prado
Guggenheim Partners, LLC; Lawrence Berkeley National Laboratory; Harvard University - RCC
August 11, 2013
A large number of quantitative hedge funds have historically sustained losses. In this study we argue that the backtesting methodology at the core of their strategy selection process may have played a role.
* Most firms and portfolio managers rely on backtests (or historical simulations of performance) to allocate capital to investment strategies.
* After trying only 7 strategy configurations, a researcher is expected to identify at least one 2-year long backtest with an annualized Sharpe ratio of over 1, when the expected out of sample Sharpe ratio is 0.
* If the researcher tries a large enough number of strategy configurations, a backtest can always be fit to any desired performance for a fixed sample length. Thus, there is a minimum backtest length (MinBTL) that should be required for a given number of trials.
* Standard statistical techniques designed to prevent regression over-fitting, such as hold-out, are inaccurate in the context of backtest evaluation.
* The practical totality of published backtests do not report the number of trials involved.
* Under memory effects, over-fitting leads to systematic losses, not noise.
Number of Pages in PDF File: 58
Keywords: backtest, historical simulation, probability of backtest over-fitting, investment strategy, optimization, Sharpe ratio, minimum back-test length, performance degradation
JEL Classification: G0, G1, G2, G15, G24, E44
Date posted: August 12, 2013 ; Last revised: July 5, 2015