The Future of Societal Impact Assessment Using Peer Review: Pre-Evaluation Training, Consensus Building and Inter-Reviewer Reliability

10 Pages Posted: 3 Jun 2017

See all articles by Gemma Derrick

Gemma Derrick

Lancaster University

Gabrielle Samuel

Lancaster University

Date Written: May 2017

Abstract

There are strong political reasons underpinning the desire to achieve a high level of inter-reviewer reliability (IRR) within peer review panels. Achieving a high level of IRR is synonymous with an efficient review system, and the wider perception of a fair evaluation process. Therefore, there is an arguable role for a more structured approach to the peer review process during a time when evaluators are effectively novices in practice with the criterion, such as with societal impact. This article explores the consequences of a structured peer review process that aimed to increase inter-reviewer reliability within panels charged with assessing societal impact. Using a series of interviews from evaluators from the UK’s Research Excellence Framework conducted before (pre-evaluation) and then again after the completion of the process (post-evaluation), it explores evaluators’ perceptions about how one tool of a structured evaluation process, pre-evaluation training, influenced their approaches to achieving a consensus within the peer review panel. Building on lessons learnt from studies on achieving inter-reviewer reliability and from consensus building with peer review groups, this article debates the benefits of structured peer review processes in cases when the evaluators are unsure of the criterion (as was the case with the Impact criterion), and therefore the risks of a low IRR are increased. In particular, this article explores how individual approaches to assessing Impact were normalized during group deliberation around Impact and how these relate to evaluators’ perceptions of the advice given during the pre-evaluation training. This article is published as part of a collection on the future of research assessment.

Suggested Citation

Derrick, Gemma and Samuel, Gabrielle, The Future of Societal Impact Assessment Using Peer Review: Pre-Evaluation Training, Consensus Building and Inter-Reviewer Reliability (May 2017). Palgrave Communications, Vol. 3, 2017. Available at SSRN: https://ssrn.com/abstract=2973198 or http://dx.doi.org/10.1057/palcomms.2017.40

Gemma Derrick (Contact Author)

Lancaster University ( email )

Lancaster LA1 4YX
United Kingdom

Gabrielle Samuel

Lancaster University ( email )

Lancaster LA1 4YX
United Kingdom

Here is the Coronavirus
related research on SSRN

Paper statistics

Downloads
15
Abstract Views
236
PlumX Metrics