The Revealed Expertise Algorithm: Leveraging Advice-taking to Identify Experts and Improve Wisdom of Crowds

69 Pages Posted: 29 Jan 2021 Last revised: 13 Sep 2022

See all articles by Yunhao Zhang

Yunhao Zhang

Massachusetts Institute of Technology (MIT) - Sloan School of Management

Date Written: November 29, 2020

Abstract

Identifying the experts within a crowd may help further improve the wisdom of crowds. I propose a new Revealed Expertise (RE) algorithm that uses the "RE measure", which is a scaled amount of belief updating given numerical advice (i.e., the group mean), as a proxy for prior variance to better reflect the relative expertise of each agent in a crowd. The intuition, which we confirm both theoretically and empirically, is that those who are less swayed by the group mean tend to be more accurate in their initial judgment. Therefore, using inverse-variance weighting with the RE measures as the variance inputs outperforms the existing wisdom-of-crowds methods by over-weighting the more accurate initial judgments in the aggregation. Crucially, we demonstrate that while self-reported confidence reflects one's feeling of uncertainty given one's available information, advice-taking reveals the amount of information one has and has not taken into account in their initial judgment. Therefore, the RE algorithm is able to successfully identify the experts, even when self-reported confidence fails. In addition, I show that the RE algorithm improves Wisdom of Crowds in a context where people might be biased (e.g., right-leaning Republicans and left-leaning Democrats answer political trivia questions). Nevertheless, I still propose a pre-registered method in which we measure subjects' bias in advice-taking to calibrate the RE measures and further improve the algorithm's performance.

Keywords: Judgment Aggregation, Wisdom of Crowds, Uncertainty, Belief-update

JEL Classification: D79, D89

Suggested Citation

Zhang, Yunhao, The Revealed Expertise Algorithm: Leveraging Advice-taking to Identify Experts and Improve Wisdom of Crowds (November 29, 2020). Available at SSRN: https://ssrn.com/abstract=3739192 or http://dx.doi.org/10.2139/ssrn.3739192

Yunhao Zhang (Contact Author)

Massachusetts Institute of Technology (MIT) - Sloan School of Management ( email )

100 Main Street
E62-416
Cambridge, MA 02142
United States

HOME PAGE: http://https://mitsloan.mit.edu/phd/students/yunhao-zhang

Do you have a job opening that you would like to promote on SSRN?

Paper statistics

Downloads
162
Abstract Views
692
Rank
280,920
PlumX Metrics