Consumers Believe Star Ratings are Comparable Even When They are Not

Abstract

User-generated ratings–often elicited and presented as “star ratings”–are among the most influential sources of information for consumers online, but do they help consumers make utility-maximizing decisions? We argue that ratings are created with respect to a frame of reference the specific product engenders, including expectations for quality/experience. As such, any rating is relative to this context. Because different products can elicit different frames of reference, there is no guarantee that ratings are comparable across alternatives. To this end, we show that inferior products can receive higher ratings than their objectively better competition. This would not be an issue if consumers recognize the role of context when interpreting reviews, however our evidence suggests they do not. Across five laboratory studies and an analysis of over 400,000 observations from Airbnb.com, we find that products receive lower ratings when expectations are higher, that this can cause discordance between differences in ratings and differences in objective quality, and that consumers are biased towards choosing the higher rated product even when it is demonstrably worse.

Keywords: user-generated ratings, comparability, evaluability, judgments, online ratings