Towards Measuring Journal Impact—Properly
Center for the History of Political Economy at Duke University Working Paper Series
20 Pages Posted:
Date Written: December 23, 2020
In this paper I evaluate the Journal Impact Factor using a theory of measurement. I argue that JIF does not stand up to close scrutiny. To measure a concept adequately, our theory of measurement requires correspondence between three steps: the characterisation of the concept, its representation, and the procedures followed to carry out the measurement. Characterisation involves defining the concept: identifying its boundaries, which fixes the features that belong to it. Representation involves defining a metrical system that appropriately represents the concept. The procedures are the rules formulated for applying the metrical system to the tokens. These three steps do not line up together neatly for JIF. There are at least two problems. First, the procedures to measure JIF do not reflect an unequivocal characterisation. Second, the representation strategy of JIF is inappropriate and not justified, given the kind of concept it tries to capture: one without strict boundaries. The bottom line is not that the JIF ought to be eschewed. Sufficient reasons related to how JIF distorts scientists’ incentives have been provided to this end. But path dependence is a tricky issue—the longevity of qwerty keyboards demonstrates it. The bottom line is that, given that JIF is unlikely to vanish, we better start giving it some proper scientific basis.
Suggested Citation: Suggested Citation