The Probability of Backtest Overfitting

Abstract

Many investment firms and portfolio managers rely on backtests (ie, simulations of performance based on historical market data) to select investment strategies and allocate capital. Standard statistical techniques designed to prevent regression overfitting, such as hold-out, tend to be unreliable and inaccurate in the context of investment backtests. We propose a general framework to assess the probability of backtest overfitting (PBO).We illustrate this framework with specific generic, model-free and nonparametric implementations in the context of investment simulations; we call these implementations combinatorially symmetric cross-validation (CSCV). We show that CSCV produces reasonable estimates of PBO for several useful examples.

Keywords: backtest, overfitting, investment strategy, Sharpe ratio optimization, performance degradation