Citation Counts: Are They Good Predictors of Rae Scores? A Bibliometric Analysis of RAE 2001

36 Pages Posted: 2 Jul 2008

See all articles by Surya Mahdi

Surya Mahdi

Cranfield School of Management

Pablo d'Este

Cranfield School of Management

Andy Neely

University of Cambridge; London Business School

Date Written: March 1, 2008

Abstract

The question of how best to assess research performance is clearly of great concern. In December 2007, HEFCE launched a national consultation on the future of Research Assessment, proposing that a Research Excellence Framework replaces the current Research Assessment Exercise. Fundamentally the Research Excellence Framework involves a shift to metrics. Views on the effectiveness of metrics for assessing research are mixed, so in this report we seek to explore empirically the question of whether metrics based on citation counts are strongly correlated with peer review assessments of research quality. We use data from RAE 2001, covering all departments and all universities in the UK. At a more disaggregated level the data used in this study includes: [i] the individual submissions made to RAE 2001 - a database of 203,743 research output records, one for each output submitted in RAE 2001; [ii] the citation counts for each of the submitted items when they are journal articles (141,789 of the original 203,743 items). We have interrogated the ISA Web of Science and produced citation counts for every article included in RAE 2001 where data is available. This gives us citation counts for 112,201 publications (55.1% of the original 203,743 submitted items); [iii] the actual RAE 2001 scores produced and published by HEFCE. We contrast these scores - which are based on peer review - with those that would have been produced had bibliometrics, based on citation counts, been used in RAE 2001.

The findings of this analysis are: [i] citation counts are a reasonable proxy for peer assessment in some subjects, such as Biological Sciences, Clinical Sciences, Chemistry and Psychology; [ii] however, citation counts are a weak proxy for a large number of disciplines, including fields within Biomedical related subjects and Engineering-related subjects. This is despite the fact that these subjects have good coverage in the Web of Science, in terms of the outputs submitted to RAE 2001; [iii] as others have reported, the Web of Science offers only partial coverage in some subjects, and hence the use of bibliometrics becomes increasingly less valuable as we move from Biomedical, Physical Sciences and Engineering, to Socials Sciences, Literature and Arts and Humanities.

Keywords: Performance, productivity, innovation, science, RAE

Suggested Citation

Mahdi, Surya and d'Este, Pablo and Neely, Andy D., Citation Counts: Are They Good Predictors of Rae Scores? A Bibliometric Analysis of RAE 2001 (March 1, 2008). Available at SSRN: https://ssrn.com/abstract=1154053 or http://dx.doi.org/10.2139/ssrn.1154053

Surya Mahdi

Cranfield School of Management ( email )

United Kingdom

Pablo D'Este

Cranfield School of Management ( email )

United Kingdom

Andy D. Neely (Contact Author)

University of Cambridge ( email )

Trinity Ln
Cambridge, CB2 1TN
United Kingdom

HOME PAGE: http://www.ifm.eng.cam.ac.uk/people/adn1000/

London Business School ( email )

Sussex Place
Regent's Park
London, London NW1 4SA
United Kingdom