Identify Experts through Revealed Confidence: An Application in Wisdom of Crowds

71 Pages Posted: 29 Jan 2021 Last revised: 26 Dec 2021

See all articles by Yunhao Zhang

Yunhao Zhang

Massachusetts Institute of Technology (MIT) - Sloan School of Management

Date Written: November 29, 2020

Abstract

Identifying the experts within a crowd may help further improve the wisdom of crowds.
However, conventional confidence elicitation, such as subjective confidence intervals, can not
reliably identify the experts. A new Revealed Confidence (RC) algorithm is proposed, which
uses a scaled amount of belief-updating given a numerical advice (e.g., the group mean) as
a measure of uncertainty, to better reflect the relative expertise of each agent. I develop a
semi-Bayesian belief-updating model to show that RC reveals both first-order uncertainty (e.g.,
the width of the confidence interval) and second-order uncertainty (e.g., uncertainty about the
width of the confidence interval). Empirical studies test, and confirm, several of the predictions
of this model: (1) RC is able to improve upon the existing wisdom of crowd methods by
overweighting the relatively more accurate answers in the aggregation; (2) laypersons with
less information might report a narrower confidence interval than the more informed experts,
resulting in the failure of confidence intervals to identify the experts, yet RC is able to correctly
identify the experts even so, because RC reflects the amount of information one has or has
not initially taken into account; (3) the optimal advice used in the RC algorithm needs to be
set reasonably distant from the agents’ initial answers. This condition is naturally satisfied
when the actual group mean is distant from the initial answers. Only when this condition is
not satisfied, I propose artificially setting an advice that is distant from each initial answer to
achieve better RC performance.

Keywords: Judgment Aggregation, Wisdom of Crowds, Uncertainty, Belief-update

JEL Classification: D79, D89

Suggested Citation

Zhang, Yunhao, Identify Experts through Revealed Confidence: An Application in Wisdom of Crowds (November 29, 2020). Available at SSRN: https://ssrn.com/abstract=3739192 or http://dx.doi.org/10.2139/ssrn.3739192

Yunhao Zhang (Contact Author)

Massachusetts Institute of Technology (MIT) - Sloan School of Management ( email )

100 Main Street
E62-416
Cambridge, MA 02142
United States

HOME PAGE: http://https://mitsloan.mit.edu/phd/students/yunhao-zhang

Do you want regular updates from SSRN on Twitter?

Paper statistics

Downloads
77
Abstract Views
439
rank
415,946
PlumX Metrics