Scoring Expert Forecasts

5 Pages Posted: 2 Jun 2017

See all articles by Kenneth C. Lichtendahl

Kenneth C. Lichtendahl

University of Virginia - Darden School of Business

Yael Grushka-Cockayne

University of Virginia - Darden School of Business; Harvard University - Business School (HBS)

Abstract

This technical note, based on the more comprehensive note, "Eliciting and Evaluating Expert Forecasts" (UVA-QA-0734), provides a streamlined presentation of Brier and log scores as tools for assessing forecasting records among a pool of experts. The note is designed to be used in conjunction with a forecasting exercise.

Excerpt

UVA-QA-0772

Rev. Sept. 12, 2014

Scoring Expert FORECASTS

Evaluating the forecasts of others can be a difficult task. One approach is to score an expert's forecast once the realization of the uncertainty is known. A track record of high scores on multiple forecasts may yield important insights into the expertise an individual possesses. In this note, we describe several scoring rules for evaluating expert opinion.

Scoring Forecasts of Discrete Events

Scoring rules first appeared in the 1950s to evaluate meteorological forecasts. Since that time, scoring rules have found a wide variety of applications in business and other fields. To this day, meteorologists in the United States are evaluated using a Brier scoring rule. When a discrete uncertainty has only two possible outcomes (e.g., rain/no rain), the Brier scoring rule assigns a score of –(1 – p)2, where p is the probability forecast reported for the event that occurs.

. . .

Keywords: log scoring rule, Brier scoring rule, fractile, quantile, percentile, probability mass function, probability density function

Suggested Citation

Lichtendahl, Kenneth C. and Grushka-Cockayne, Yael, Scoring Expert Forecasts. Darden Case No. UVA-QA-0772. Available at SSRN: https://ssrn.com/abstract=2975140

Kenneth C. Lichtendahl (Contact Author)

University of Virginia - Darden School of Business ( email )

P.O. Box 6550
Charlottesville, VA 22906-6550
United States

Yael Grushka-Cockayne

University of Virginia - Darden School of Business ( email )

P.O. Box 6550
Charlottesville, VA 22906-6550
United States

Harvard University - Business School (HBS) ( email )

Soldiers Field Road
Morgan 270C
Boston, MA 02163
United States

HOME PAGE: http://https://www.hbs.edu/faculty/Pages/profile.aspx?facId=263650

Register to save articles to
your library

Register

Paper statistics

Downloads
0
Abstract Views
86
PlumX Metrics