University of California, Los Angeles (UCLA); National Bureau of Economic Research (NBER)
October 22, 2012
This paper analyzes referee recommendations in two settings: The first setting is a prestigious finance conference, in which a computer algorithm matched referees to papers based only on shared expertise. The second setting is the standard journal process, with data from eight prominent economics and finance journals (ECMTA, JEEA, JET, QJE, IER, RAND, JF, RFS). Despite referee selection differences, the data suggest similar referee behavior in both settings. First, referees display only modest consensus. Second, referees disagree not only about scales (a referee mean effect), but also about the relative ordering of papers. Third, the bias measured by average generosity of the referee on other papers is about as important in predicting a referee’s recommendation as the opinion of another referee on the same paper.
In sum, the typical referee report consists roughly of one part signal of some referee-agreeable objective attribute of the paper and two parts (referee-specific) noise. In turn, the noise itself consists roughly of one part referee-mean effect (bias) and two parts unidentified effects or noise.
Number of Pages in PDF File: 47
JEL Classification: A14working papers series
Date posted: August 29, 2012 ; Last revised: October 23, 2012
© 2013 Social Science Electronic Publishing, Inc. All Rights Reserved.
This page was processed by apollo5 in 0.562 seconds