Finding the Wise and the Wisdom in a Crowd: Simultaneously Estimating the Underlying Qualities of Raters and Items from a Series of Reviews
63 Pages Posted: 30 Sep 2019 Last revised: 24 Feb 2020
Date Written: September 15, 2019
Consumers and businesses rely on others' ratings of items when making choices. However, individual reviewers vary in their accuracy, and some are biased -- either systematically over- or under-rating items relative to others' tastes, or even deliberately distorting a rating. We provide a new technique that processes ratings by a group of reviewers over a set of items and simultaneously evaluates the individual reviewers' accuracies and biases, and provides unbiased and efficient estimates of the items' true qualities. We show that our technique generates significant improvements over an average of ratings in recovering qualities even with small data sets, and that this improvement increases as the number of items increases. Revisiting the famous 1976 wine tasting that compared Californian and Bordeaux wines, we find substantial variation in reviewers' accuracies and a ranking that differs from the original one based on average ratings. In addition, we apply our methodology to more than forty-five thousand ratings of Bordeaux wines. Our estimated wine qualities significantly predict prices when controlling for prominent experts' ratings and numerous fixed effects. We also find that the elasticity of a wine price in an expert's ratings increases with that expert's accuracy.
Keywords: Ratings, Experts, Reviews, Qualities, Scoring, Aggregating Ratings, Wine Ratings, Wine, Bias
JEL Classification: D80, D82, C13, C15, Q10
Suggested Citation: Suggested Citation