The Anatomy of Out-of-Sample Forecasting Accuracy

54 Pages Posted: 16 Nov 2022

See all articles by Daniel Borup

Daniel Borup

Aarhus University, CREATES, DFI

Philippe Goulet Coulombe

University of Quebec at Montreal (UQAM) - Université du Québec à Montréal

David Rapach

Research Department, Federal Reserve Bank of Atlanta; Washington University in St. Louis

Erik Christian Montes Schütte

Aarhus University; Aarhus University - CREATES; DFI

Sander Schwenk-Nebbe

Aarhus University - Department of Economics and Business Economics

Date Written: November 16, 2022

Abstract

We develop metrics based on Shapley values for interpreting time-series forecasting models, including “black-box” models from machine learning. Our metrics are model agnostic, so that they are applicable to any model (linear or nonlinear, parametric or nonparametric). Two of the metrics, iShapley-VI and oShapley-VI, measure the importance of individual predictors in fitted models for explaining the in-sample and out-of-sample predicted target values, respectively. The third metric is the performance-based Shapley value (PBSV), our main methodological contribution. PBSV measures the contributions of individual predictors in fitted models to the out-of-sample loss and thereby anatomizes out-of-sample forecasting accuracy. In an empirical application forecasting US inflation, we find important discrepancies between individual predictor relevance according to the in-sample iShapley-VI and out-of-sample PBSV. We use simulations to analyze potential sources of the discrepancies, including overfitting, structural breaks, and evolving predictor volatilities.

Keywords: variable importance, out-of-sample performance, Shapley value, loss function, machine learning, inflation

JEL Classification: C22, C45, C53, E37, G17

Suggested Citation

Borup, Daniel and Coulombe, Philippe Goulet and Rapach, David and Schütte, Erik Christian Montes and Schütte, Erik Christian Montes and Schwenk-Nebbe, Sander, The Anatomy of Out-of-Sample Forecasting Accuracy (November 16, 2022). FRB Atlanta Working Paper No. 2022-16, https://doi.org/10.29338/wp2022-16, Available at SSRN: https://ssrn.com/abstract=4278745 or http://dx.doi.org/10.2139/ssrn.4278745

Daniel Borup

Aarhus University, CREATES, DFI ( email )

School of Business and Social Sciences
Fuglesangs Alle 4
Aarhus V, 8210
Denmark

Philippe Goulet Coulombe

University of Quebec at Montreal (UQAM) - Université du Québec à Montréal

CP 8888 Succ Centre Ville
Montréal, Québec H3C3P8
Canada

David Rapach (Contact Author)

Research Department, Federal Reserve Bank of Atlanta ( email )

1000 Peachtree Street N.E.
Atlanta, GA 30309-4470
United States

Washington University in St. Louis

One Brookings Drive
Campus Box 1133
St. Louis, MO 63130-4899
United States

HOME PAGE: http://https://sites.google.com/slu.edu/daverapach

Erik Christian Montes Schütte

Aarhus University ( email )

Nordre Ringgade 1
DK-8000 Aarhus C, 8000
Denmark

Aarhus University - CREATES ( email )

School of Economics and Management
Building 1322, Bartholins Alle 10
DK-8000 Aarhus C
Denmark

HOME PAGE: http://sites.google.com/view/christian-montes-schutte/home

DFI ( email )

Sander Schwenk-Nebbe

Aarhus University - Department of Economics and Business Economics ( email )

Fuglesangs Allé 4
Aarhus V, 8210
Denmark

Do you have a job opening that you would like to promote on SSRN?

Paper statistics

Downloads
140
Abstract Views
493
rank
302,487
PlumX Metrics