A discussion paper for possible approaches to building a statistically valid backtesting framework

Abstract

This paper explores potential methodologies for constructing a backtesting framework for financial institutions that is statistically valid. Although backtesting is an indispensable instrument for ensuring regulatory compliance and risk management, existing practices in the industry have certain deficiencies that compromise their credibility. Commencing with an examination of prevailing industry methodologies of backtesting, we underscore their deficiencies. Following this, the theoretical foundations of statistical validity are elucidated, and the constraints of present methodologies are deliberated. Following this, we present several alternatives for constructing a statistically valid backtesting framework, assessing their prospective advantages and disadvantages. In summary, we provide suggestions for the execution of a statistically valid backtesting framework as well as potential avenues for future investigation.

Keywords: Data snooping, evaluation bias, e-values, overfitting, p-hacking