Boosting the Wisdom of Crowds Within a Single Judgment Problem: Selective Averaging Based on Peer Predictions
26 Pages Posted: 1 Jan 2020 Last revised: 15 Jun 2020
Date Written: June 12, 2020
Combining point estimates from multiple judges often provides a more accurate consensus estimate than using a point estimate from a single judge, a phenomenon called "the wisdom of crowds." This consensus can often be improved further by averaging estimates from a carefully chosen subset of judges, known as a select crowd. However, previously proposed approaches for deciding which and how many judges to include in the select crowd rely on past performance data from similar questions. This paper develops methodology to identify a high-performing subset of judges within a single estimation problem. Judges are asked to provide both a point estimate of the quantity of interest and a prediction of the average estimate that will be given by all other judges. Predictions of others are then used as part of a customized criterion to select which estimates to include in the consensus for that problem. Our selection procedure is robust to noise in the judges’ responses and can be solved quickly even for a large crowd. We use both simulated and experimental data to illustrate that the procedure can improve the accuracy of the consensus estimate.
Keywords: Forecasting, Estimation, Judgment Aggregation, Wisdom of Crowds, Shared Information
JEL Classification: D83, C53, C91
Suggested Citation: Suggested Citation