The Anatomy of Out-of-Sample Forecasting Accuracy

36 Pages Posted: 23 Feb 2024

See all articles by Daniel Borup

Daniel Borup

Aarhus University, CREATES, DFI

Philippe Goulet Coulombe

Université du Québec à Montréal - Département des Sciences Économiques

David Rapach

Research Department, Federal Reserve Bank of Atlanta; Washington University in St. Louis

Erik Christian Montes Schütte

Aarhus University; Aarhus University - CREATES; DFI

Sander Schwenk-Nebbe

Aarhus University - Department of Economics and Business Economics

Multiple version iconThere are 2 versions of this paper

Date Written: February, 2024

Abstract

We introduce the performance-based Shapley value (PBSV) to measure the contributions of individual predictors to the out-of-sample loss for time-series forecasting models. Our new metric allows a researcher to anatomize out-of-sample forecasting accuracy, thereby providing valuable information for interpreting time-series forecasting models. The PBSV is model agnostic—so it can be applied to any forecasting model, including "black box" models in machine learning, and it can be used for any loss function. We also develop the TS-Shapley-VI, a version of the conventional Shapley value that gauges the importance of predictors for explaining the in-sample predictions in the entire sequence of fitted models that generates the time series of out-of-sample forecasts. We then propose the model accordance score to compare predictor ranks based on the TS-Shapley-VI and PBSV, thereby linking the predictors' in-sample importance to their contributions to out-of-sample forecasting accuracy. We illustrate our metrics in an application forecasting US inflation.

Keywords: model interpretation, Shapley value, predictor importance, loss function, machine learning, inflation

JEL Classification: C22, C45, C52, C53, E31, E37

Suggested Citation

Borup, Daniel and Goulet Coulombe, Philippe and Rapach, David and Schütte, Erik Christian Montes and Schütte, Erik Christian Montes and Schwenk-Nebbe, Sander, The Anatomy of Out-of-Sample Forecasting Accuracy (February, 2024). FRB Atlanta Working Paper No. 2022-16B, Available at SSRN: https://ssrn.com/abstract=4736378 or http://dx.doi.org/10.29338/wp2022-16b

Daniel Borup

Aarhus University, CREATES, DFI ( email )

School of Business and Social Sciences
Fuglesangs Alle 4
Aarhus V, 8210
Denmark

Philippe Goulet Coulombe

Université du Québec à Montréal - Département des Sciences Économiques ( email )

PB 8888 Station DownTown
Succursale Centre Ville
Montreal, Quebec H3C3P8
Canada

David Rapach (Contact Author)

Research Department, Federal Reserve Bank of Atlanta ( email )

1000 Peachtree Street N.E.
Atlanta, GA 30309-4470
United States

Washington University in St. Louis ( email )

One Brookings Drive
Campus Box 1133
St. Louis, MO 63130-4899
United States

HOME PAGE: http://https://sites.google.com/slu.edu/daverapach

Erik Christian Montes Schütte

Aarhus University ( email )

Nordre Ringgade 1
DK-8000 Aarhus C, 8000
Denmark

Aarhus University - CREATES ( email )

School of Economics and Management
Building 1322, Bartholins Alle 10
DK-8000 Aarhus C
Denmark

HOME PAGE: http://sites.google.com/view/christian-montes-schutte/home

DFI ( email )

Sander Schwenk-Nebbe

Aarhus University - Department of Economics and Business Economics ( email )

Fuglesangs Allé 4
Aarhus V, 8210
Denmark

Do you have a job opening that you would like to promote on SSRN?

Paper statistics

Downloads
126
Abstract Views
569
Rank
62,102
PlumX Metrics