Expert judgments versus publication-based metrics: Do the two methods produce identical results in measuring academic reputation?

Abstract

The purpose of this study is to assess the validity of citation metrics based on disciplinary representative surveys. The present project compared citation rankings for individual scientists with expert judgments collected through a survey of 818 Russian sociologists. We have used scientometric indicators, not limited to traditional metrics, but including special indicators measuring non-standard publication output and potential metric abuse. In gathering bibliometric indicators, we rely on the national bibliometric data source that is especially important for non-Western higher education systems. Our study showed that scientometric indicators predict the names of the most influential sociologists with significant accuracy. Citation metrics and survey results converge to a large extent, at least concerning the goal of identifying the disciplinary elite. We found two types of deviations: false positive and false negative identifications. False-positive identifications are explained by specialization in writing widely cited textbooks and the explicit gaming of metrics; these can, however, be identified.

Keywords: citation metrics, academic reputation, validity of citation rankings, expert judgements