Boosting the Wisdom of Crowds Within a Single Judgment Problem: Selective Averaging Based on Peer Predictions

26 Pages Posted: 1 Jan 2020 Last revised: 15 Jun 2020

See all articles by Asa Palley

Asa Palley

Indiana University - Kelley School of Business - Department of Operation & Decision Technologies

Ville Satopää

INSEAD - Technology and Operations Management

Date Written: June 12, 2020

Abstract

Combining point estimates from multiple judges often provides a more accurate consensus estimate than using a point estimate from a single judge, a phenomenon called "the wisdom of crowds." This consensus can often be improved further by averaging estimates from a carefully chosen subset of judges, known as a select crowd. However, previously proposed approaches for deciding which and how many judges to include in the select crowd rely on past performance data from similar questions. This paper develops methodology to identify a high-performing subset of judges within a single estimation problem. Judges are asked to provide both a point estimate of the quantity of interest and a prediction of the average estimate that will be given by all other judges. Predictions of others are then used as part of a customized criterion to select which estimates to include in the consensus for that problem. Our selection procedure is robust to noise in the judges’ responses and can be solved quickly even for a large crowd. We use both simulated and experimental data to illustrate that the procedure can improve the accuracy of the consensus estimate.

Keywords: Forecasting, Estimation, Judgment Aggregation, Wisdom of Crowds, Shared Information

JEL Classification: D83, C53, C91

Suggested Citation

Palley, Asa and Satopää, Ville, Boosting the Wisdom of Crowds Within a Single Judgment Problem: Selective Averaging Based on Peer Predictions (June 12, 2020). Available at SSRN: https://ssrn.com/abstract=3504286 or http://dx.doi.org/10.2139/ssrn.3504286

Asa Palley (Contact Author)

Indiana University - Kelley School of Business - Department of Operation & Decision Technologies ( email )

Hodge Hall 4100
1275 E 10th St.
Bloomington, IN 47405
United States

Ville Satopää

INSEAD - Technology and Operations Management ( email )

Boulevard de Constance
77 305 Fontainebleau Cedex
France

Here is the Coronavirus
related research on SSRN

Paper statistics

Downloads
34
Abstract Views
230
PlumX Metrics