General Properties of Backtestable Statistics
30 Pages Posted: 25 Jan 2017
Date Written: January 24, 2017
We propose a formal definition of backtestable statistic: a backtest is a null expected value involving only the statistic and its random variable, strictly monotonic in the former. We discuss the relationship with elicitability and identifiability which turn out being necessary conditions. The variance and the Expected Shortfall are not backtestable for this reason. We discuss (absolute) model validation in the context of one– or two–sided hypothesis tests, as well as (relative) model selection obtained by ranking realizations of the backtest statistic. We introduce the concept of sharpness which refers to whether a backtest is strictly monotonic with respect to the real value of the statistic and not only to its prediction. This decides whether the expected value of a backtest determines the extent of a prediction discrepancy and not only its likelihood. We show that the quantile backtest is not sharp and in fact provides no information whatsoever on the real value of the statistic. The Expectile is also not sharp; we provide bounds for its real value, which are looser for outer confidence levels. We then introduce ridge backtests, applicable to particular non–backtestable statistics, such as the variance and the Expected Shortfall, which coincide with the attained minimum of the scoring function of another elicitable auxiliary statistic. This allows to produce sharp backtest procedures in which the prediction for the auxiliary variable is also involved but with small sensitivity and known bias sign. The ridge mechanism explains why the variance has always been de–facto backtestable and allows for similar efficient ways to backtest the expected shortfall. We discuss the relevance of this result in the current debate of financial regulation (banking and insurance), where Value at Risk and Expected Shortfall are adopted as regulatory risk measures.
Keywords: Backtesting, Quantile, Value at Risk, Expected Shortfall, Expectile, Elicitability
JEL Classification: D81, G32
Suggested Citation: Suggested Citation