Limitations of Quantitative Claims About Trading Strategy Evaluation

Abstract

One of the key assumptions of quantitative trading strategy evaluation is that Type II
errors (missed discoveries) are preferable to Type I errors (false discoveries). However,
practitioners have known for a long time that the statistical properties of some genuine
trading strategies are often indistinguishable from those of random trading strategies.
Therefore, any adjustments to statistics to guard against p-hacking increase Type II error
unless the power of the test is high. At the same time, the power of the test is limited by
insufficient samples and changing market conditions. Furthermore, genuine strategies
with statistical properties that are similar to those of random strategies may overfit due to
favorable market conditions but fail when market conditions change. These facts severely
limit the effectiveness of quantitative claims about trading strategy evaluation.
Practitioners have instead resorted to Monte Carlo simulations and stochastic modeling in
an effort to increase the chances of identifying robust trading strategies, but these
methods also have severe limitations due to changing market conditions, selection bias,
and data snooping. In this paper, we present two examples that demonstrate the
limitations of quantitative evaluation of trading strategies, and we claim that the most
effective way of guarding against overfitting and selection bias is by limiting the
applications of backtesting to a class of strategies that employ similar but simple
predictors of price. We claim that determining when market conditions change is, in
many cases, fundamentally more important than any quantitative claims about trading
strategy evaluation.

Keywords: Trading strategy, data mining, market timing, moving averages, performance evaluation

JEL Classification: G10, G17