Performance Evaluation: Subjectivity, Bias and Judgement Style in Sport Data

Abstract

The number of situations that require individual judgements and evaluations, and that can be object of different sources of conscious and unconscious biases is endless. This paper proposes a practical scores aggregation procedure that attempts to reduce and mitigate the influence of bias in subjective judgements. The argument is based on the idea that bias is associated with deviations from the panel mean and/or deviations from the judge grading style. Consequently, the procedure is not specific to a particular type of bias (e.g., nationalistic or reputation bias), but it addresses general forms of bias. We also discuss a set of desirable properties. The proposed scores aggregation procedure is then applied to a unique data set from the 2000 Summer Olympic Games diving competition. The proposed procedure captures the possibility of bias by adjusting the gap of scores between divers.

Keywords: Scores aggregation; Subjective judgements; Biased judgements; Bias correction; Sports data

JEL Classification: D71, D72, D91