Decomposing the Effects of Crowd-Wisdom Aggregators: The Bias-Information-Noise (BIN) Model

Abstract

Aggregating predictions from multiple judges often yields more accurate predictions than relying on a single judge: the wisdom-of-the-crowd effect. But there is a wide range of aggregation methods, from one-size- fits-all techniques, such as simple averaging, prediction markets, and Bayesian aggregators to customized (supervised) techniques, such as weighted averaging, that require past performance data. This article applies a wide range of aggregation methods to subjective probability estimates from geopolitical forecasting tournaments. It uses the Bias-Information-Noise (BIN) model to disentangle three mechanisms by which aggregators improve accuracy: the tamping down of bias and noise and the extraction of valid information across forecasters. Simple averaging works almost entirely by reducing noise, whereas more complex techniques, like prediction markets and Bayesian aggregators, work via all three pathways: better signal extraction as well as noise and bias reduction. We close by exploring the utility of a BIN approach to the modular construction of aggregators.

Keywords: Bayesian Statistics, Judgmental Forecasting, Partial Information, Prediction Markets, Wisdom of Crowds