The Revealed Expertise Algorithm: Leveraging Advice-taking to Identify Experts and Improve Wisdom of Crowds

Abstract

Identifying the experts within a crowd may help further improve the wisdom of crowds. I propose a new Revealed Expertise (RE) algorithm that uses the "RE measure", which is a scaled amount of belief updating given numerical advice (i.e., the group mean), as a proxy for prior variance to better reflect the relative expertise of each agent in a crowd. The intuition, which we confirm both theoretically and empirically, is that those who are less swayed by the group mean tend to be more accurate in their initial judgment. Therefore, using inverse-variance weighting with the RE measures as the variance inputs outperforms the existing wisdom-of-crowds methods by over-weighting the more accurate initial judgments in the aggregation. Crucially, we demonstrate that while self-reported confidence reflects one's feeling of uncertainty given one's available information, advice-taking reveals the amount of information one has and has not taken into account in their initial judgment. Therefore, the RE algorithm is able to successfully identify the experts, even when self-reported confidence fails. In addition, I show that the RE algorithm improves Wisdom of Crowds in a context where people might be biased (e.g., right-leaning Republicans and left-leaning Democrats answer political trivia questions). Nevertheless, I still propose a pre-registered method in which we measure subjects' bias in advice-taking to calibrate the RE measures and further improve the algorithm's performance.

Keywords: Judgment Aggregation, Wisdom of Crowds, Uncertainty, Belief-update

JEL Classification: D79, D89