Editorial Decisions and Measuring Information Contents of Reviews

Abstract

I measure information on distinct facets of quality from a corpus of reviews and characterize how decision-makers integrate this information present in text with that available through other channels. Specifically, I demonstrate that referee comments at a scholarly journal contain information on submissions’ future citation impact above and beyond information available in referee scores. I measure this signal on future citation impact and show that it does not enter into editorial decision-making directly but rather through an interaction that amplifies the information content of referee scores: the more citations a low- or mediocre-scoring paper is likely to get the less likely it is to be published. Secondly, I describe referee comments that are highly predictive of greater citations. Papers that referees say have access to unique datasets, or are written on topics of relevance to ongoing debates or government applications receive greater citations on average. Third, I show the appearance of favoritism amongst editors who accept a higher share of papers that cite themselves is partly a reflection of an ability to draw and select for papers that receive more citations. Finally, I characterize budget constraints on publication space and referee capital and provide some guidance on what types of information editorial systems could capture to promote transparency in future analyses while protecting privacy of authors or referees.

Keywords: influence, signaling, editorial decisions, referee reviews, information asymmetries, collective decision-making, textual analysis, journals, R&D, innovation, scientific commons

JEL Classification: A14, C81, D0, D2, D71, D8