Score-Based Calibration Testing for Multivariate Forecast Distributions

41 Pages Posted: 7 Feb 2023

See all articles by Malte Knüppel

Malte Knüppel

Deutsche Bundesbank

Marc-Oliver Pohle

Goethe University Frankfurt

Fabian Krüger

Karlsruhe Institute of Technology

Date Written: 2022

Abstract

Multivariate distributional forecasts have become widespread in recent years. To assess the quality of such forecasts, suitable evaluation methods are needed. In the univariate case, calibration tests based on the probability integral transform (PIT) are routinely used. However, multivariate extensions of PIT-based calibration tests face various challenges. We therefore introduce a general framework for calibration testing in the multivariate case and propose two new tests that arise from it. Both approaches use proper scoring rules and are simple to implement even in large dimensions. The first employs the PIT of the score. The second is based on comparing the expected performance of the forecast distribution (i.e., the expected score) to its actual performance based on realized observations (i.e., the realized score). The tests have good size and power properties in simulations and solve various problems of existing tests. We apply the new tests to forecast distributions for macroeconomic and financial time series data.

Keywords: Forecast Evaluation, Density Forecasts, Ensemble Forecasts

JEL Classification: C12, C52, C53

Suggested Citation

Knüppel, Malte and Pohle, Marc-Oliver and Krüger, Fabian, Score-Based Calibration Testing for Multivariate Forecast Distributions (2022). Deutsche Bundesbank Discussion Paper No. 50/2022, Available at SSRN: https://ssrn.com/abstract=4350792 or http://dx.doi.org/10.2139/ssrn.4350792

Malte Knüppel (Contact Author)

Deutsche Bundesbank

Wilhelm-Epstein-Str. 14
Frankfurt/Main, 60431
Germany

Marc-Oliver Pohle

Goethe University Frankfurt ( email )

Grüneburgplatz 1
Frankfurt am Main, 60323
Germany

Fabian Krüger

Karlsruhe Institute of Technology ( email )

Kaiserstraße 12
Karlsruhe, Baden Württemberg 76131
Germany

Do you have negative results from your research you’d like to share?

Paper statistics

Downloads
21
Abstract Views
140
PlumX Metrics