Trimmed Opinion Pools and the Crowd's Calibration Problem

Management Science, Forthcoming

Darden Business School Working Paper No. 2175093

22 Pages Posted: 15 Nov 2012 Last revised: 13 Jul 2013

See all articles by Victor Richmond Jose

Victor Richmond Jose

Georgetown University - McDonough School of Business

Yael Grushka-Cockayne

University of Virginia - Darden School of Business; Harvard University - Business School (HBS)

Kenneth C. Lichtendahl

University of Virginia - Darden School of Business

Date Written: November 13, 2012

Abstract

We introduce an alternative to the popular linear opinion pool for combining individual probability forecasts. One of the well-known problems with the linear opinion pool is that it can be poorly calibrated. It tends toward underconfidence as the crowd’s diversity increases, i.e., as the variance in the individuals’ means increases. To address this calibration problem, we propose the exterior-trimmed opinion pool. To form this pool, forecasts with low and high means, or cumulative distribution function (cdf) values, are trimmed away from a linear opinion pool. Exterior-trimming decreases the pool’s variance and improves its calibration. A linear opinion pool, however, will remain overconfident when individuals are overconfident and not very diverse. For these situations, we suggest trimming away forecasts with moderate means, or cdf values. This interior-trimming increases variance and reduces overconfidence. Using probability forecast data from US and European Surveys of Professional Forecasters, we present empirical evidence that trimmed opinion pools can outperform the linear opinion pool.

Keywords: trimming, probability forecasts, expert combination, linear opinion pool, underconfidence, overconfidence, scoring rules, wisdom of crowds, diversity

JEL Classification: C10, C53, E17

Suggested Citation

Jose, Victor Richmond and Grushka-Cockayne, Yael and Lichtendahl, Kenneth C., Trimmed Opinion Pools and the Crowd's Calibration Problem (November 13, 2012). Management Science, Forthcoming; Darden Business School Working Paper No. 2175093. Available at SSRN: https://ssrn.com/abstract=2175093 or http://dx.doi.org/10.2139/ssrn.2175093

Victor Richmond Jose

Georgetown University - McDonough School of Business ( email )

544 Hariri Bldg
37th and O Sts NW
Washington, DC 20057
United States

Yael Grushka-Cockayne (Contact Author)

University of Virginia - Darden School of Business ( email )

P.O. Box 6550
Charlottesville, VA 22906-6550
United States

Harvard University - Business School (HBS) ( email )

Soldiers Field Road
Morgan 270C
Boston, MA 02163
United States

HOME PAGE: http://https://www.hbs.edu/faculty/Pages/profile.aspx?facId=263650

Kenneth C. Lichtendahl

University of Virginia - Darden School of Business ( email )

P.O. Box 6550
Charlottesville, VA 22906-6550
United States

Register to save articles to
your library

Register

Paper statistics

Downloads
177
Abstract Views
1,445
rank
173,576
PlumX Metrics