Identify Experts through Revealed Confidence: An Application in Wisdom of Crowds

Abstract

Identifying the experts within a crowd may help further improve the wisdom of crowds.
However, conventional confidence elicitation, such as subjective confidence intervals, can not
reliably identify the experts. A new Revealed Confidence (RC) algorithm is proposed, which
uses a scaled amount of belief-updating given a numerical advice (e.g., the group mean) as
a measure of uncertainty, to better reflect the relative expertise of each agent. I develop a
semi-Bayesian belief-updating model to show that RC reveals both first-order uncertainty (e.g.,
the width of the confidence interval) and second-order uncertainty (e.g., uncertainty about the
width of the confidence interval). Empirical studies test, and confirm, several of the predictions
of this model: (1) RC is able to improve upon the existing wisdom of crowd methods by
overweighting the relatively more accurate answers in the aggregation; (2) laypersons with
less information might report a narrower confidence interval than the more informed experts,
resulting in the failure of confidence intervals to identify the experts, yet RC is able to correctly
identify the experts even so, because RC reflects the amount of information one has or has
not initially taken into account; (3) the optimal advice used in the RC algorithm needs to be
set reasonably distant from the agents’ initial answers. This condition is naturally satisfied
when the actual group mean is distant from the initial answers. Only when this condition is
not satisfied, I propose artificially setting an advice that is distant from each initial answer to
achieve better RC performance.

Keywords: Judgment Aggregation, Wisdom of Crowds, Uncertainty, Belief-update

JEL Classification: D79, D89