Quantile Evaluation, Sensitivity to Bracketing, and Sharing Business Payoffs

30 Pages Posted: 9 Jul 2015 Last revised: 7 Nov 2016

See all articles by Yael Grushka-Cockayne

Yael Grushka-Cockayne

University of Virginia - Darden School of Business; Harvard University - Business School (HBS)

Kenneth C. Lichtendahl

University of Virginia - Darden School of Business

Victor Richmond Jose

Georgetown University - McDonough School of Business

Robert L. Winkler

Duke University - Fuqua School of Business

Date Written: November 6, 2016

Abstract

From forecasting competitions to conditional value-at-risk requirements, the use of multiple quantile assessments is growing in practice. To evaluate them, we use a rule from the general class of proper scoring rules for a forecaster's multiple quantiles of a single uncertain quantity of interest. The general rule is additive in the component scores. Each component contains a function that measures its quantile's distance from the realization and weights its contribution to the overall score. To determine this function, we propose that the score of a group's combined quantile should be better than that of a randomly-selected forecaster's quantile only when the forecasters bracket the realization, or their quantiles do not all fall on the same side of the realization. If a score satisfies this property, we say it is sensitive to bracketing. We characterize the class of proper scoring rules that is sensitive to bracketing when the decision maker uses a generalized average to combine forecasters' quantiles. Finally, we show how weights can be set to match the payoffs in many important business contexts.

Keywords: quantile forecasts; forecast evaluation, probability elicitation, proper scoring rules, expert aggregation, expert combination

Suggested Citation

Grushka-Cockayne, Yael and Lichtendahl, Kenneth C. and Jose, Victor Richmond and Winkler, Robert L., Quantile Evaluation, Sensitivity to Bracketing, and Sharing Business Payoffs (November 6, 2016). Available at SSRN: https://ssrn.com/abstract=2628322 or http://dx.doi.org/10.2139/ssrn.2628322

Yael Grushka-Cockayne (Contact Author)

University of Virginia - Darden School of Business ( email )

P.O. Box 6550
Charlottesville, VA 22906-6550
United States

Harvard University - Business School (HBS) ( email )

Soldiers Field Road
Morgan 270C
Boston, MA 02163
United States

HOME PAGE: http://https://www.hbs.edu/faculty/Pages/profile.aspx?facId=263650

Kenneth C. Lichtendahl

University of Virginia - Darden School of Business ( email )

P.O. Box 6550
Charlottesville, VA 22906-6550
United States

Victor Richmond Jose

Georgetown University - McDonough School of Business ( email )

544 Hariri Bldg
37th and O Sts NW
Washington, DC 20057
United States

Robert L. Winkler

Duke University - Fuqua School of Business ( email )

Box 90120
Durham, NC 27708-0120
United States

Register to save articles to
your library

Register

Paper statistics

Downloads
95
Abstract Views
1,243
rank
276,784
PlumX Metrics