Extending the Wisdom of Crowds: Quantifying Uncertainty Using the Mean and Variance of a Collection of Point Estimates

Abstract

The wisdom of crowds—combining information from a collection of individual judgments—offers a useful technique to quantify an unknown variable. Averaging point estimates has proven to be effective in reducing error in the consensus estimate. However, in many managerial problems, the decision maker requires an assessment of the full distribution of uncertainty rather than just a single number. In practice, some managers have used dispersion in point estimates as a cue to uncertainty in the variable of interest, but a characterization of the exact relationship between the two has not yet been established. Using a stylized Bayesian model of overlapping information spread across a collection of judges, I show that the variance of the variable of interest can be estimated with a multiple of the variance of the individual judgments, and establish an analytical expression for the consensus predictive distribution. I then present a procedure that can be used to learn about this variance inflation factor using past judgments and realizations, and derive the resulting distribution for the new variable of interest. This aggregation method is easy to implement in practice to estimate a predictive distribution from a collection of individual judgments. Application of the procedure to forecasts available through the Survey of Professional Forecasters suggests that the resulting predictions are well-calibrated and outperform natural existing alternatives.

Keywords: Judgment Aggregation, Wisdom of Crowds, Forecasting, Combining Information, Estimation

JEL Classification: D83, C53, C91