Decomposing the Effects of Crowd-Wisdom Aggregators: The Bias-Information-Noise (BIN) Model

25 Pages Posted: 19 Mar 2021

See all articles by Ville Satopää

Ville Satopää

INSEAD - Technology and Operations Management

Marat Salikhov

Yale School of Management

Philip Tetlock

University of Pennsylvania

Barb Mellers

University of Pennsylvania, Psychology; University of Pennsylvania, Wharton School

Date Written: February 8, 2021

Abstract

Aggregating predictions from multiple judges often yields more accurate predictions than relying on a single judge: the “wisdom-of-the-crowd” effect. This aggregation can be conducted by different methods, from simple averaging to complex techniques, like Bayesian estimators and prediction markets. This article applies a broad set of aggregation methods to subjective probability estimates from a series of geopolitical forecasting tournaments. It then uses the Bias-Information-Noise (BIN) model to disentangle three mechanisms by which each aggregation method improves accuracy: the tamping down of bias and noise and the extraction of valid information across forecasters. Averaging works almost entirely via noise reduction whereas more complex techniques, like prediction markets and Bayesian aggregators, work via all three BIN pathways: better signal extraction and noise and bias reduction.

Keywords: Bayesian Statistics, Judgmental Forecasting, Partial Information, Prediction Markets, Wisdom of Crowds

Suggested Citation

Satopää, Ville and Salikhov, Marat and Tetlock, Philip and Mellers, Barb, Decomposing the Effects of Crowd-Wisdom Aggregators: The Bias-Information-Noise (BIN) Model (February 8, 2021). Available at SSRN: https://ssrn.com/abstract=3781405 or http://dx.doi.org/10.2139/ssrn.3781405

Ville Satopää (Contact Author)

INSEAD - Technology and Operations Management ( email )

Boulevard de Constance
77 305 Fontainebleau Cedex
France

Marat Salikhov

Yale School of Management ( email )

165 Whitney Ave
New Haven, CT 06511

HOME PAGE: http://maratsalikhov.com

Philip Tetlock

University of Pennsylvania ( email )

Philadelphia, PA 19104
United States

Barb Mellers

University of Pennsylvania, Psychology ( email )

3815 Walnut Street
Philadelphia, PA 19104-6196
United States

University of Pennsylvania, Wharton School ( email )

3641 Locust Walk
Philadelphia, PA 19104-6365
United States

Do you have a job opening that you would like to promote on SSRN?

Paper statistics

Downloads
375
Abstract Views
1,241
rank
96,167
PlumX Metrics