Bias in expert product reviews

Abstract

Hard evidence on bias in expert reviews and its consequences for ratings is rare. This holds particularly true for conflicts of interest that are thought to be common in non-blind product reviews but are not readily observable: ad hoc relationships between reviewers and producers. We present a textbook case of a long-running expert product review in the food service industry for which we happen to know the reviewer's conflict of interest: being affiliated to one particular producer. As is typical, only insiders were aware of the possible source of bias in the review. The review resembles other non-blind tests of product quality. We obtained detailed data to map the consequences of the conflict of interest. We find evidence of a sizable bias in the reviewers' ratings. Our findings suggest that reviewers' ad hoc relationships with producers, often dismissed as `coming with the job', can be very harmful.