When Do Experts Listen to Other Experts? The Role of Negative Information in Expert Evaluations for Novel Projects

46 Pages Posted: 20 Jul 2020 Last revised: 30 Jul 2020

See all articles by Jacqueline Lane

Jacqueline Lane

Harvard Business School

Misha Teplitskiy

University of Michigan School of Information

Gary Gray

affiliation not provided to SSRN

Hardeep Ranu

affiliation not provided to SSRN

Michael Menietti

Harvard University - Business School (HBS)

Eva Guinan

Harvard Medical School; Dana-Farber Cancer Institute

Karim R. Lakhani

Harvard Business School - Technology and Operations Management Group; Harvard Institute for Quantitative Social Science; Harvard University - Berkman Klein Center for Internet & Society

Date Written: July 9, 2020

Abstract

The evaluation of novel projects lies at the heart of scientific and technological innovation, and yet literature suggests that this process is subject to inconsistency and potential biases. This paper investigates the role of information sharing among experts as the driver of evaluation decisions. We designed and executed two field experiments in two separate grant funding opportunities at a leading research university to explore evaluators’ receptivity to assessments from other evaluators. Collectively, our experiments mobilized 369 evaluators from seven universities to evaluate 97 projects resulting in 760 proposal-evaluation pairs and over $300,000 in awards. We exogenously varied two key aspects of information sharing: 1) the intellectual distance between each focal evaluator and the other evaluators and 2) the relative valence (positive and negative) of others’ scores, to determine how these treatments affect the focal evaluator’s propensity to change the initial score. Although the intellectual similarity treatment did not yield a measurable effect, we found causal evidence of negativity bias, where evaluators are more likely to lower their scores after seeing critical scores than raise them after seeing better scores. Qualitative coding and topic modeling of the evaluators’ justifications for score changes reveal that exposures to low scores prompted greater attention to uncovering weaknesses, whereas exposures to neutral or high scores were associated with strengths, along with greater emphasis on non-evaluation criteria, such as confidence in one’s judgment. Overall, information sharing among expert evaluators can lead to more conservative allocation decisions that favors protecting against failure than maximizing success.

Keywords: project evaluation, innovation, knowledge frontier, diversity, negativity bias

Suggested Citation

Lane, Jacqueline and Teplitskiy, Misha and Gray, Gary and Ranu, Hardeep and Menietti, Michael and Guinan, Eva and Lakhani, Karim R., When Do Experts Listen to Other Experts? The Role of Negative Information in Expert Evaluations for Novel Projects (July 9, 2020). Harvard Business School Technology & Operations Mgt. Unit Working Paper No. 21-007, Available at SSRN: https://ssrn.com/abstract=3656495 or http://dx.doi.org/10.2139/ssrn.3656495

Jacqueline Lane

Harvard Business School ( email )

Soldiers Field Road
Morgan 270C
Boston, MA 02163
United States

Misha Teplitskiy

University of Michigan School of Information ( email )

Ann Arbor, MI
United States

HOME PAGE: http://www.mishateplitskiy.com

Gary Gray

affiliation not provided to SSRN ( email )

No Address Available

Hardeep Ranu

affiliation not provided to SSRN

Michael Menietti

Harvard University - Business School (HBS) ( email )

Soldiers Field Road
Morgan 270C
Boston, MA 02163
United States

Eva Guinan

Harvard Medical School ( email )

Dana-Farber Cancer Institute
450 Brookline Avenue, Suite M1B29
Boston, MA 02215
United States
617-632-4932 (Phone)
617-632-3770 (Fax)

Dana-Farber Cancer Institute ( email )

Dana-Farber Cancer Institute
450 Brookline Avenue, Suite M1B29
Boston, MA 02215
United States
617-632-4932 (Phone)
617-632-3770 (Fax)

Karim R. Lakhani (Contact Author)

Harvard Business School - Technology and Operations Management Group ( email )

Boston, MA 02163
United States
617-495-6741 (Phone)

Harvard Institute for Quantitative Social Science ( email )

1737 Cambridge St.
Cambridge, MA 02138
United States

Harvard University - Berkman Klein Center for Internet & Society ( email )

Harvard Law School
23 Everett, 2nd Floor
Cambridge, MA 02138
United States

Here is the Coronavirus
related research on SSRN

Paper statistics

Downloads
87
Abstract Views
527
rank
322,023
PlumX Metrics