Evaluating Probability Forecasts for GDP Declines Using Alternative Methodologies

Abstract

Evaluation methodologies for rare events from meteorology, psychology and medical diagnosis are used to examine the value of probability forecasts of real GDP declines during the current (Q0) and each of the next four quarters (Q1-Q4) using data from the Survey of Professional Forecasters. We study the quality of these probability forecasts in terms of calibration, resolution, odds ratio, the relative operating characteristic (ROC), and alternative variance decompositions. Only the shorter-term forecasts (Q0-Q2) are found to possess significant skill in terms of all measures considered, even though they are characterized by excess variability and lack of calibration.

The battery of diagnostic statistics cataloged in this paper should be useful for evaluating regression models with binary dependent variables, particularly when the event of interest is relatively uncommon.

Keywords: Binary prediction, Rare events, Survey of Professional Forecasters, Subjective probability, Calibration, Resolution, Skill score, Relative Operating Characteristics, Odds ratio, Recession

JEL Classification: B22, C11, C22, C53