Abstract

https://ssrn.com/abstract=2745220
 


 



All that Glitters Is Not Gold: Comparing Backtest and Out-of-Sample Performance on a Large Cohort of Trading Algorithms


Thomas Wiecki


Quantopian Inc

Andrew Campbell


Quantopian Inc.

Justin Lent


Quantopian Inc

Jessica Stauth


Quantopian Inc

March 9, 2016


Abstract:     
When automated trading strategies are developed and evaluated using backtests on historical pricing data, there exists a tendency to overfit to the past. Using a unique dataset of 888 algorithmic trading strategies developed and backtested on the Quantopian platform with at least 6 months of out-of-sample performance, we study the prevalence and impact of backtest overfitting. Specifically, we find that commonly reported backtest evaluation metrics like the Sharpe ratio offer little value in predicting out of sample performance (R² < 0.025). In contrast, higher order moments, like volatility and maximum drawdown, as well as portfolio construction features, like hedging, show significant predictive value of relevance to quantitative finance practitioners. Moreover, in line with prior theoretical considerations, we find empirical evidence of overfitting – the more backtesting a quant has done for a strategy, the larger the discrepancy between backtest and out-of-sample performance. Finally, we show that by training non-linear machine learning classifiers on a variety of features that describe backtest behavior, out-of-sample performance can be predicted at a much higher accuracy (R² = 0.17) on hold-out data compared to using linear, univariate features. A portfolio constructed on predictions on hold-out data performed significantly better out-of-sample than one constructed from algorithms with the highest backtest Sharpe ratios.

Number of Pages in PDF File: 19

Keywords: quantitative finance, algorithmic trading, backtesting, overfitting, machine learning


Open PDF in Browser Download This Paper

Date posted: April 9, 2016  

Suggested Citation

Wiecki, Thomas and Campbell, Andrew and Lent, Justin and Stauth, Jessica, All that Glitters Is Not Gold: Comparing Backtest and Out-of-Sample Performance on a Large Cohort of Trading Algorithms (March 9, 2016). Available at SSRN: https://ssrn.com/abstract=2745220 or http://dx.doi.org/10.2139/ssrn.2745220

Contact Information

Thomas Wiecki (Contact Author)
Quantopian Inc ( email )
100 Franklin Street
Boston, MA 02110
United States
Andrew Campbell
Quantopian Inc. ( email )
100 Franklin Street
Boston, MA 02110
United States
Justin Lent
Quantopian Inc ( email )
100 Franklin Street
Boston, MA 02110
United States
Jessica Stauth
Quantopian Inc ( email )
100 Franklin Street
Boston, MA 02110
United States
Feedback to SSRN


Paper statistics
Abstract Views: 8,797
Downloads: 3,391
Download Rank: 2,048