Algorithm-Human-Algorithm: A New Classification Approach to Integrating Judgemental Adjustments

Abstract

Firms often elicit judgemental adjustments to an algorithm-generated demand forecast. This process aims to utilize humans’ private information that is inaccessible to the algorithm. However, humans are vulnerable to systematic biases when making such adjustments. Thus, it is challenging to integrate these adjustments with algorithm-generated forecasts to improve forecast accuracy (measured as the absolute deviation from realized sales). We propose a novel classification-based solution to address this challenge. First, we predict an adjustment’s quality (likelihood to improve forecast accuracy) using predictors of humans’ private information advantage (e.g., product and store characteristics) and systematic biases (e.g., recent forecast errors and adjustment characteristics). Next, we apply a simple heuristic -- based on the predicted quality -- to classify (accept/reject) each adjustment integration in the final forecast.

We collaborate with a European retailer to test our approach using a large dataset (~ 1.1mn transactions) of algorithm-generated forecasts. Humans adjusted 38% of these forecasts, and nearly 51% of these adjustments improved forecast accuracy. In out-of-sample tests, we benchmark our approach against the strategy of using historical evidence of humans’ private information advantage to determine when to always accept or reject adjustments at the product-store level. We find that our approach improves benchmark forecast accuracy by 12%. Moreover, a substantial share (46%) of this improvement is associated with the inclusion of predictors of systematic biases.

Our paper expands practitioners' toolkit for managing judgemental adjustments by showcasing an intuitive and easy-to-implement adjustment-level integration solution.

Keywords: Judgemental Adjustments, Human, Algorithm, Demand Forecasting, Systematic Bias, Machine Learning