Confidence Calibration in a Multiyear Geopolitical Forecasting Competition

Management Science 63, no. 11 (2017): 3552-3565.

43 Pages Posted: 6 Aug 2020

See all articles by Don A. Moore

Don A. Moore

University of California, Berkeley - Haas School of Business

Samuel A. Swift

Carnegie Mellon University

Angela Minster

affiliation not provided to SSRN

Barbara Mellers

affiliation not provided to SSRN

Lyle Ungar

University of Pennsylvania

Philip Tetlock

University of Pennsylvania

Heather Yang

Massachusetts Institute of Technology (MIT) - Sloan School of Management

Elizabeth R. Tenney

University of Virginia - Psychology

Date Written: March 18, 2016

Abstract

This research examines the development of confidence and accuracy over time in the context of forecasting. Although overconfidence has been studied in many contexts, little research examines its progression over long periods of time or in consequential policy domains. This study employs a unique data set from a geopolitical forecasting tournament spanning three years in which thousands of forecasters predicted the outcomes of hundreds of events. We sought to apply insights from research to structure the questions, interactions, and elicitation to improve forecasts. Indeed, forecasters’ confidence roughly matched their accuracy. As information came in, accuracy increased. Confidence increased at approximately the same rate as accuracy, and good calibration persisted. Nevertheless, there was evidence of a small amount of overconfidence (3%), especially on the most confident forecasts. Training helped reduce overconfidence and team collaboration improved forecast accuracy. Together, teams and training reduced overconfidence to 1%. Our results provide reason for tempered optimism regarding confidence calibration and its development over time in consequential field contexts.

Keywords: confidence, overconfidence, forecasting, prediction

Suggested Citation

Moore, Don A. and Swift, Samuel A. and Minster, Angela and Mellers, Barbara and Ungar, Lyle and Tetlock, Philip and Yang, Heather and Tenney, Elizabeth R., Confidence Calibration in a Multiyear Geopolitical Forecasting Competition (March 18, 2016). Management Science 63, no. 11 (2017): 3552-3565., Available at SSRN: https://ssrn.com/abstract=3643605

Don A. Moore (Contact Author)

University of California, Berkeley - Haas School of Business ( email )

545 Student Services Building, #1900
2220 Piedmont Avenue
Berkeley, CA 94720
United States

Samuel A. Swift

Carnegie Mellon University ( email )

Pittsburgh, PA 15213-3890
United States

Angela Minster

affiliation not provided to SSRN

Barbara Mellers

affiliation not provided to SSRN

Lyle Ungar

University of Pennsylvania

Philadelphia, PA 19104
United States

Philip Tetlock

University of Pennsylvania ( email )

Philadelphia, PA 19104
United States

Heather Yang

Massachusetts Institute of Technology (MIT) - Sloan School of Management ( email )

100 Main Street
E62-416
Cambridge, MA 02142
United States

Elizabeth R. Tenney

University of Virginia - Psychology ( email )

1400 University Ave
Charlottesville, VA 22903
United States

Here is the Coronavirus
related research on SSRN

Paper statistics

Downloads
8
Abstract Views
45
PlumX Metrics