Is It Better To Elicit Quantile Or Probability Judgments to Estimate a Continuous Distribution? Elicit Both and Create an Inner Crowd.

48 Pages Posted: 8 Jun 2017 Last revised: 6 May 2021

See all articles by Asa Palley

Asa Palley

Indiana University - Kelley School of Business - Department of Operation & Decision Technologies

Saurabh Bansal

Pennsylvania State University - Smeal College of Business

Date Written: May 6, 2021

Abstract

Managers frequently rely on the judgment of an expert to estimate a probability distribution for a continuous random variable. Two elicitation methods are commonly used to gather this information: (i) quantile judgments for a set of fixed probability values, or (ii) probability judgments based on a set of fixed variable values, but a consensus on which format yields more accurate distribution estimates has not been reached. We report results from a series of eight experiments conducted with 1,456 participants with a range of quantitative backgrounds to compare these elicitation formats for a variety of variables, including synthetically generated numbers displayed in a video, daily high and low temperatures, the ages of people with a given name, commute times, home prices, and household incomes. In total, our data set includes 30,870 complete distribution judgments. The data show that probability distributions constructed from quantile judgments tend to be more accurate than those obtained from cumulative probability judgments. However, visual probability elicitation tools offered similar performance to quantile judgments, with each format holding a slight advantage according to different metrics. On average, quantile judgments were slightly further from the empirical distribution while visual probability judgments were slightly overconfident. We find that averaging the distributions obtained from quantile judgments with those obtained from visual probability judgments provides superior performance to either approach, delivering significantly better accuracy and calibration of confidence intervals. Based on these results, we recommend, when possible, eliciting judgments using both the quantile and visual probability formats and combining the resulting distributions.

Keywords: Elicitation, Subjective Probability Distribution, Expert Judgment, Estimation

JEL Classification: D83, C53, C91

Suggested Citation

Palley, Asa and Bansal, Saurabh, Is It Better To Elicit Quantile Or Probability Judgments to Estimate a Continuous Distribution? Elicit Both and Create an Inner Crowd. (May 6, 2021). Kelley School of Business Research Paper No. 17-44, Available at SSRN: https://ssrn.com/abstract=2981840 or http://dx.doi.org/10.2139/ssrn.2981840

Asa Palley (Contact Author)

Indiana University - Kelley School of Business - Department of Operation & Decision Technologies ( email )

Hodge Hall 4100
1275 E 10th St.
Bloomington, IN 47405
United States

Saurabh Bansal

Pennsylvania State University - Smeal College of Business ( email )

University Park, PA 16802
United States

Do you have a job opening that you would like to promote on SSRN?

Paper statistics

Downloads
141
Abstract Views
1,244
rank
277,282
PlumX Metrics