Ensembles of Overfit and Overconfident Forecasts

37 Pages Posted: 3 Aug 2014 Last revised: 19 Aug 2015

See all articles by Yael Grushka-Cockayne

Yael Grushka-Cockayne

University of Virginia - Darden School of Business; Harvard University - Business School (HBS)

Victor Richmond Jose

Georgetown University - McDonough School of Business

Kenneth C. Lichtendahl

University of Virginia - Darden School of Business

Date Written: August 18, 2015

Abstract

Firms today average forecasts collected from multiple experts and models. Because of cognitive biases, strategic incentives, or the structure of machine-learning algorithms, these forecasts are often overfit to sample data and are overconfident. Little is known about the challenges associated with aggregating such forecasts. We introduce a theoretical model to examine the combined effect of overfitting and overconfidence on the average forecast. Their combined effect is that the mean and median probability forecasts are poorly calibrated with hit rates of their prediction intervals too high and too low, respectively. Consequently, we prescribe the use of a trimmed average, or trimmed opinion pool, to achieve better calibration. We identify the random forest, a leading machine-learning algorithm that pools hundreds of overfit and overconfident regression trees, as an ideal environment for trimming probabilities. Using several known datasets, we demonstrate that trimmed ensembles can significantly improve the random forest's predictive accuracy.

Keywords: wisdom of crowds; base-rate neglect; linear opinion pool; trimmed opinion pool; hit rate; calibration; random forest.

JEL Classification: C10, C53, E17

Suggested Citation

Grushka-Cockayne, Yael and Jose, Victor Richmond and Lichtendahl, Kenneth C., Ensembles of Overfit and Overconfident Forecasts (August 18, 2015). Darden Business School Working Paper No. 2474438. Available at SSRN: https://ssrn.com/abstract=2474438 or http://dx.doi.org/10.2139/ssrn.2474438

Yael Grushka-Cockayne (Contact Author)

University of Virginia - Darden School of Business ( email )

P.O. Box 6550
Charlottesville, VA 22906-6550
United States

Harvard University - Business School (HBS) ( email )

Soldiers Field Road
Morgan 270C
Boston, MA 02163
United States

HOME PAGE: http://https://www.hbs.edu/faculty/Pages/profile.aspx?facId=263650

Victor Richmond Jose

Georgetown University - McDonough School of Business ( email )

544 Hariri Bldg
37th and O Sts NW
Washington, DC 20057
United States

Kenneth C. Lichtendahl

University of Virginia - Darden School of Business ( email )

P.O. Box 6550
Charlottesville, VA 22906-6550
United States

Register to save articles to
your library

Register

Paper statistics

Downloads
345
Abstract Views
1,959
rank
88,587
PlumX Metrics