The Revealed Expertise Algorithm: Leveraging Advice-taking to Identify Experts and Improve Wisdom of Crowds

Abstract

Identifying the experts within a crowd may help further improve the wisdom of crowds. We propose a new Revealed Expertise (RE) algorithm that uses the "RE measure", which is a scaled amount of belief updating given numerical advice (i.e., the group mean), as a proxy for prior variance to better reflect the relative expertise of each agent in a crowd. The intuition, which we confirm empirically, is that those who are less swayed by the group mean tend to be more accurate in their initial judgment. Therefore, using inverse-variance weighting with the RE measures as the variance inputs improves upon the existing wisdom-of-crowd methods by over-weighting the more accurate initial judgments in the aggregation. Crucially, we demonstrate that advice-taking is able to reveal the amount of information one has and has not taken into account in their initial judgment, even when self-reported confidence mistakes the less informed as the experts. In addition, we propose a pre-registered method in which we measure subjects' bias in advice-taking to calibrate the RE measures and further improve the algorithm's performance.

Keywords: Judgment Aggregation, Wisdom of Crowds, Uncertainty, Belief-update

JEL Classification: D79, D89