Backtesting Correlated Quantities
19 Pages Posted: 26 Sep 2023 Last revised: 11 Apr 2024
Date Written: September 13, 2023
Abstract
Backtesting financial models over long horizons inevitably leads to correlated samples due to overlapping windows. As a result, the null distributions of popular test statistics like exceedence counting or chi-squared are no longer analytically given. A standard technique to solve this problem is to calculate these distributions numerically via Monte Carlo simulation. While this automatically accounts for the correlation in the samples, the resulting distributions can sometimes have very long tails, which results in significant loss of discriminatory power of the test. We introduce a simple yet effective pre-processing technique based on decorrelating the samples, hence making them compatible with standard statistical tests. We provide various numerical examples showing that this leads to more stable distributions, higher discriminatory power and natural generalizations to jointly backtesting multiple correlated quantities in a clean consistent framework.
Keywords: Backtesting, CCR, statistics, hypothesis test, discriminatory power, model validation
JEL Classification: C12, C15, C22, C52
Suggested Citation: Suggested Citation