The Revealed Confidence Algorithm: Leveraging Advice-taking to Identify Experts and Improve Wisdom of Crowds

Abstract

Identifying the experts within a crowd may help further improve the wisdom of crowds, especially when the crowd is biased. We propose a novel Revealed Confidence (RC) algorithm that employs the “RC measure” to better reflect the relative expertise of each agent in a crowd. This RC measure is the scaled amount of one’s belief-updating in response to numerical advice. The intuition, which we confirm both theoretically and empirically, is that those who are less swayed by numerical advice (e.g., the group mean) tend to be more accurate in their initial judgment. Therefore, taking a weighted average over the initial judgments with weights inversely proportional to the respective RC measures would place greater weight on the more accurate initial judgments in the aggregation. Consequently, the RC algorithm outperforms the conventional wisdom-of-crowds methods by overweighting the experts’ judgments. Moreover, we find a causal relationship between the amount of information one has and the degree of advice-taking in an experimental setting – experts (agents who have more information) are less swayed by numerical advice. Nevertheless, a higher stated confidence (e.g., a narrower confidence interval) does not correspond to having more information or being an expert. Since expertise is measured by the amount of information one has (and has not) taken into account, the RC algorithm may effectively distinguish the experts and improve Wisdom of Crowds, even when the stated confidence interval fails to do so.

Keywords: Judgment Aggregation, Wisdom of Crowds, Subjective uncertainty, Advice-taking, Belief-updating

JEL Classification: D79, D89