Probabilistic Coherence Weighting for Optimizing Expert Forecasts

Decision Analysis, 10(4), 305-326

23 Pages Posted: 21 Mar 2014  

Christopher W. Karvetski

George Mason University

Kenneth C. Olson

George Mason University

David R. Mandel

York University - Department of Psychology

Charles R. Twardy

George Mason University

Date Written: July 26, 2013

Abstract

Methods for eliciting and aggregating expert judgment are necessary when decision-relevant data are scarce. Such methods have been used for aggregating the judgments of a large, heterogeneous group of forecasters, as well as the multiple judgments produced from an individual forecaster. This paper addresses how multiple related individual forecasts can be used to improve aggregation of probabilities for a binary event across a set of forecasters. We extend previous efforts that use probabilistic incoherence of an individual forecaster’s subjective probability judgments to weight and aggregate the judgments of multiple forecasters for the goal of increasing the accuracy of forecasts. With data from two studies, we describe an approach for eliciting extra probability judgments to (i) adjust the judgments of each individual forecaster, and (ii) assign weights to the judgments to aggregate over the entire set of forecasters. We show improvement of up to 30% over the established benchmark of a simple equal-weighted averaging of forecasts. We also describe how this method can be used to remedy the “fifty–fifty blip” that occurs when forecasters use the probability value of 0.5 to represent epistemic uncertainty.

Keywords: probabilistic coherence, forecast aggregation, crowdsourcing, linear opinion pool, fifty–fifty blip, practice

Suggested Citation

Karvetski, Christopher W. and Olson, Kenneth C. and Mandel, David R. and Twardy, Charles R., Probabilistic Coherence Weighting for Optimizing Expert Forecasts (July 26, 2013). Decision Analysis, 10(4), 305-326. Available at SSRN: https://ssrn.com/abstract=2411649

Christopher W. Karvetski

George Mason University ( email )

4400 University Drive
Fairfax, VA 22030
United States

Kenneth C. Olson

George Mason University ( email )

4400 University Drive
Fairfax, VA 22030
United States

David R. Mandel (Contact Author)

York University - Department of Psychology ( email )

4700 Keele St.
Toronto, Ontario M3J 1P3
Canada

HOME PAGE: http://https://sites.google.com/site/themandelian/

Charles R. Twardy

George Mason University ( email )

4400 University Drive
Fairfax, VA 22030
United States

Paper statistics

Downloads
27
Abstract Views
169