How Should Research Performance Be Measured? A Study of Swedish Economists

Abstract

The increased international exchange and competition in modern economics research has boosted the demand for objective and internationally comparable evaluations of institutions and individual researchers. There exist several plausible measures of research performance, each capturing one or several relevant aspects, but none of them can be argued to suffice as the sole vehicle for the ranking and valuation of output.

Using a sample of the whole population of full professors of economics in one country (Sweden) we explore how the use of different measures influence outcomes in terms of the skewness and ranking of individual performances. We find large differences across all measures, but some deviate more than others. In particular, the journal ranking of Kalaitzidakis et al. 2003 (KMS), which was endorsed by the European Economic Association and has been extremely influential especially in Europe, appears to be an outlier among the available measures. Its distribution of performances is the most skewed, and its ranking of scholars corresponds the least with the rankings of the other measures.

Hence, relying on one single metric of research quality, especially one that is as extreme as KMS, is associated with a great risk given that researchers tend to adjust behavior in order to maximize the assessed relative and absolute value of their work.

Keywords: Impact of research, Ranking, Research output, Research productivity

JEL Classification: A11, A13, A14, B41