Decomposing the Effects of Crowd-Wisdom Aggregators: The Bias-Information-Noise (BIN) Model

26 Pages Posted: 19 Mar 2021 Last revised: 3 Nov 2021

See all articles by Ville Satopää

Ville Satopää

INSEAD - Technology and Operations Management

Marat Salikhov

Yale School of Management

Philip Tetlock

University of Pennsylvania

Barb Mellers

University of Pennsylvania, Psychology; University of Pennsylvania, Wharton School

Date Written: February 8, 2021

Abstract

Aggregating predictions from multiple judges often yields more accurate predictions than relying on a single judge: the wisdom-of-the-crowd effect. But there is a wide range of aggregation methods, from one-size- fits-all techniques, such as simple averaging, prediction markets, and Bayesian aggregators to customized (supervised) techniques, such as weighted averaging, that require past performance data. This article applies a wide range of aggregation methods to subjective probability estimates from geopolitical forecasting tournaments. It uses the Bias-Information-Noise (BIN) model to disentangle three mechanisms by which aggregators improve accuracy: the tamping down of bias and noise and the extraction of valid information across forecasters. Simple averaging works almost entirely by reducing noise, whereas more complex techniques, like prediction markets and Bayesian aggregators, work via all three pathways: better signal extraction as well as noise and bias reduction. We close by exploring the utility of a BIN approach to the modular construction of aggregators.

Keywords: Bayesian Statistics, Judgmental Forecasting, Partial Information, Prediction Markets, Wisdom of Crowds

Suggested Citation

Satopää, Ville and Salikhov, Marat and Tetlock, Philip and Mellers, Barb, Decomposing the Effects of Crowd-Wisdom Aggregators: The Bias-Information-Noise (BIN) Model (February 8, 2021). Available at SSRN: https://ssrn.com/abstract=3781405 or http://dx.doi.org/10.2139/ssrn.3781405

Ville Satopää (Contact Author)

INSEAD - Technology and Operations Management ( email )

Boulevard de Constance
77 305 Fontainebleau Cedex
France

Marat Salikhov

Yale School of Management ( email )

165 Whitney Ave
New Haven, CT 06511

HOME PAGE: http://maratsalikhov.com

Philip Tetlock

University of Pennsylvania ( email )

Philadelphia, PA 19104
United States

Barb Mellers

University of Pennsylvania, Psychology ( email )

3815 Walnut Street
Philadelphia, PA 19104-6196
United States

University of Pennsylvania, Wharton School ( email )

3641 Locust Walk
Philadelphia, PA 19104-6365
United States

Do you have a job opening that you would like to promote on SSRN?

Paper statistics

Downloads
626
Abstract Views
2,376
rank
59,783
PlumX Metrics