Machine-Moderated Judgmental Forecasting to Improve Prediction Accuracy

Abstract

Obtaining accurate forecasts is critical to making good decisions in many managerial problems. Forecasts may be generated by a statistical model (machine) or through human judgment. Machines are well-suited to detect patterns and regularities in existing data while humans can apply their domain-specific knowledge and reasoning to the particular problem at hand. An integrated method that leverages both sources of information uses a machine to make an initial forecast and then allows humans to update this forecast. Human judgments, however, may suffer from systematic behavioral patterns that reduce their accuracy. Based on a model of humans as imperfect Bayesian updaters, we propose a three-stage hybrid approach which adds a second machine to search for and reduce biases in the human judgmental forecast. We test the approach by applying it to 6,830 forecasts collected in two controlled laboratory experiments and to 718 internal forecasts of the probability of technical success in clinical trials from a major pharmaceutical company. We show that the machine-moderated judgmental forecasting approach can provide a substantial accuracy boost in each setting, helping a manager tap into all of the information collectively available to both machines and humans.

Keywords: Bias Correction, Extremization, Human-Machine Interaction, Probability Forecasting