Eliciting Human Judgment for Prediction Algorithms

29 Pages Posted: 15 Jun 2020

See all articles by Rouba Ibrahim

Rouba Ibrahim

University College London

Song-Hee Kim

University of Southern California - Marshall school of Business

Jordan Tong

Wisconsin School of Business

Date Written: May 20, 2020

Abstract

Even when human point forecasts are less accurate than data-based algorithm predictions, they can still help boost performance by being used as algorithm inputs. Assuming one uses human judgment indirectly in this manner, we propose changing the elicitation question from the traditional direct forecast (DF) to what we call the private information adjustment (PIA): how much the human thinks the algorithm should adjust its forecast to account for the information that only the human has. Based on a behavioral model, we theoretically prove that, when there is human random error in the forecast, eliciting the PIA leads to more accurate predictions than eliciting the DF; however, this DF-PIA gap does not exist for perfectly-consistent forecasters. The DF-PIA gap is increasing in the random error people make while incorporating public information (data that the algorithm has access to) but is decreasing in the random error that people make while incorporating private information (data that only the human has access to). In controlled experiments with students and Amazon Mechanical Turk workers, we find support for these hypotheses and demonstrate the flexibility to conduct elicitation in multiple ways to enhance performance.

Keywords: laboratory experiments, behavioral operations, random error, elicitation, forecasting, prediction, discretion, expert input, private information, judgment, aggregation

Suggested Citation

Ibrahim, Rouba and Kim, Song-Hee and Tong, Jordan, Eliciting Human Judgment for Prediction Algorithms (May 20, 2020). Available at SSRN: https://ssrn.com/abstract=3606633 or http://dx.doi.org/10.2139/ssrn.3606633

Rouba Ibrahim

University College London ( email )

1 Canada Square
London, England E145AB
United Kingdom

HOME PAGE: http://www.roubaibrahim.com

Song-Hee Kim (Contact Author)

University of Southern California - Marshall school of Business ( email )

BRI 401, 3670 Trousdale Parkway
Los Angeles, CA 90089
United States

Jordan Tong

Wisconsin School of Business ( email )

975 University Avenue
Madison, WI 53706
United States

Here is the Coronavirus
related research on SSRN

Paper statistics

Downloads
15
Abstract Views
149
PlumX Metrics