Finding the Wise and the Wisdom in a Crowd: Simultaneously Estimating the Underlying Qualities of Raters and Items

66 Pages Posted: 30 Sep 2019 Last revised: 23 Aug 2021

See all articles by Nicolas Carayol

Nicolas Carayol

University of Bordeaux, Department of economics

Matthew O. Jackson

Stanford University - Department of Economics; Santa Fe Institute

Date Written: September 15, 2019

Abstract

Consumers and businesses rely on others' ratings of items when making choices. However, individual reviewers vary in their accuracy and some are biased -- either systematically over- or under-rating items relative to others' tastes, or even deliberately distorting a rating. We describe how to process ratings by a group of reviewers over a set of items and evaluate the individual reviewers' accuracies and biases, in a way that provides unbiased and consistent estimates of the items' true qualities. We provide Monte Carlo simulations that showcase the added value of our technique even with small data sets, and we show that this improvement increases as the number of items increases. Revisiting the famous 1976 wine tasting that compared Californian and Bordeaux wines, accounting for the substantial variation in reviewers' biases and accuracies results in a ranking that differs from the original average rating. We also illustrate the power of this methodology with an application to more than forty-five thousand ratings of ``en primeur'' Bordeaux fine wines by expert critics. Those data show that our estimated wine qualities significantly predict prices when controlling for prominent experts' ratings and numerous fixed effects. We also find that the elasticity of a wine price in an expert's ratings increases with that expert's accuracy.

Keywords: Ratings, Experts, Reviews, Qualities, Scoring, Aggregating Ratings, Wine Ratings, Wine, Bias

JEL Classification: D80, D82, C13, C15, Q10

Suggested Citation

Carayol, Nicolas and Jackson, Matthew O., Finding the Wise and the Wisdom in a Crowd: Simultaneously Estimating the Underlying Qualities of Raters and Items (September 15, 2019). Available at SSRN: https://ssrn.com/abstract=3419592 or http://dx.doi.org/10.2139/ssrn.3419592

Nicolas Carayol

University of Bordeaux, Department of economics ( email )

Avenue Léon Duguit
Bordeaux, 33000
France

HOME PAGE: http://ncarayol.u-bordeaux.fr/

Matthew O. Jackson (Contact Author)

Stanford University - Department of Economics ( email )

Landau Economics Building
579 Serra Mall
Stanford, CA 94305-6072
United States
1-650-723-3544 (Phone)

HOME PAGE: http://www.stanford.edu/~jacksonm

Santa Fe Institute

1399 Hyde Park Road
Santa Fe, NM 87501
United States

Do you have a job opening that you would like to promote on SSRN?

Paper statistics

Downloads
137
Abstract Views
922
rank
266,332
PlumX Metrics