Decomposing the Effects of Crowd-Wisdom Aggregators: The Bias-Information-Noise (BIN) Model
26 Pages Posted: 19 Mar 2021 Last revised: 3 Nov 2021
Date Written: February 8, 2021
Aggregating predictions from multiple judges often yields more accurate predictions than relying on a single judge: the wisdom-of-the-crowd effect. But there is a wide range of aggregation methods, from one-size- fits-all techniques, such as simple averaging, prediction markets, and Bayesian aggregators to customized (supervised) techniques, such as weighted averaging, that require past performance data. This article applies a wide range of aggregation methods to subjective probability estimates from geopolitical forecasting tournaments. It uses the Bias-Information-Noise (BIN) model to disentangle three mechanisms by which aggregators improve accuracy: the tamping down of bias and noise and the extraction of valid information across forecasters. Simple averaging works almost entirely by reducing noise, whereas more complex techniques, like prediction markets and Bayesian aggregators, work via all three pathways: better signal extraction as well as noise and bias reduction. We close by exploring the utility of a BIN approach to the modular construction of aggregators.
Keywords: Bayesian Statistics, Judgmental Forecasting, Partial Information, Prediction Markets, Wisdom of Crowds
Suggested Citation: Suggested Citation