A Heuristic for Combining Correlated Experts When There is Little Data

Abstract

It is intuitive and theoretically sound to combine experts' forecasts based on their proven skills, while accounting for correlation among their forecast submissions. Simpler combination methods, however, that assume independence of forecasts or equal skill, have been found to be empirically robust, in particular, in settings in which there is little historical data available for assessing experts' skill. One explanation for the robust performance by simple methods is that empirical estimation of skill and of correlations introduces error, leading to worse aggregated forecasts than simpler alternatives. We offer a heuristic that accounts for skill and reduces estimation error by utilizing a common correlation factor. Our theoretical results present an optimal form for this common correlation and we offer Bayesian estimators that can be used in practice. The common correlation heuristic is shown to outperform alternative combination methods on macroeconomic and experimental forecasting where there is limited historical data.