General Properties of Backtestable Statistics

30 Pages Posted: 25 Jan 2017

Date Written: January 24, 2017

Abstract

We propose a formal definition of backtestable statistic: a backtest is a null expected value involving only the statistic and its random variable, strictly monotonic in the former. We discuss the relationship with elicitability and identifiability which turn out being necessary conditions. The variance and the Expected Shortfall are not backtestable for this reason. We discuss (absolute) model validation in the context of one– or two–sided hypothesis tests, as well as (relative) model selection obtained by ranking realizations of the backtest statistic. We introduce the concept of sharpness which refers to whether a backtest is strictly monotonic with respect to the real value of the statistic and not only to its prediction. This decides whether the expected value of a backtest determines the extent of a prediction discrepancy and not only its likelihood. We show that the quantile backtest is not sharp and in fact provides no information whatsoever on the real value of the statistic. The Expectile is also not sharp; we provide bounds for its real value, which are looser for outer confidence levels. We then introduce ridge backtests, applicable to particular non–backtestable statistics, such as the variance and the Expected Shortfall, which coincide with the attained minimum of the scoring function of another elicitable auxiliary statistic. This allows to produce sharp backtest procedures in which the prediction for the auxiliary variable is also involved but with small sensitivity and known bias sign. The ridge mechanism explains why the variance has always been de–facto backtestable and allows for similar efficient ways to backtest the expected shortfall. We discuss the relevance of this result in the current debate of financial regulation (banking and insurance), where Value at Risk and Expected Shortfall are adopted as regulatory risk measures.

Keywords: Backtesting, Quantile, Value at Risk, Expected Shortfall, Expectile, Elicitability

JEL Classification: D81, G32

Suggested Citation

Acerbi, Carlo and Szekely, Balazs, General Properties of Backtestable Statistics (January 24, 2017). Available at SSRN: https://ssrn.com/abstract=2905109 or http://dx.doi.org/10.2139/ssrn.2905109

Carlo Acerbi (Contact Author)

MSCI Inc. ( email )

Geneva
Switzerland

Balazs Szekely

MSCI Inc. ( email )

Budapest
Hungary

Do you have a job opening that you would like to promote on SSRN?

Paper statistics

Downloads
862
Abstract Views
2,591
rank
41,461
PlumX Metrics