Deep Regression Ensembles

29 Pages Posted: 8 Mar 2022

See all articles by Antoine Didisheim

Antoine Didisheim

Swiss Finance Institute, UNIL

Bryan T. Kelly

Yale SOM; AQR Capital Management, LLC; National Bureau of Economic Research (NBER)

Semyon Malamud

Ecole Polytechnique Federale de Lausanne; Centre for Economic Policy Research (CEPR); Swiss Finance Institute

Date Written: March 3, 2022

Abstract

We introduce a methodology for designing and training deep neural networks (DNN) that we call 'Deep Regression Ensembles' (DRE). It bridges the gap between DNN and two-layer neural networks trained with random feature regression. Each layer of DRE has two components, randomly drawn input weights and output weights trained myopically (as if the final output layer) using linear ridge regression. Within a layer, each neuron uses a different subset of inputs and a different ridge penalty, constituting an ensemble of random feature ridge regressions. Our experiments show that a single DRE architecture is at par with or exceeds state-of-the-art DNN in many data sets. Yet, because DRE neural weights are either known in closed-form or randomly drawn, its computational cost is orders of magnitude smaller than DNN.

Keywords: Deep learning, Neural network, Random features, Ensembles

Suggested Citation

Didisheim, Antoine and Kelly, Bryan T. and Malamud, Semyon, Deep Regression Ensembles (March 3, 2022). Swiss Finance Institute Research Paper No. 22-20, 2022, Available at SSRN: https://ssrn.com/abstract=4049493 or http://dx.doi.org/10.2139/ssrn.4049493

Antoine Didisheim

Swiss Finance Institute, UNIL ( email )

c/o University of Geneva
40, Bd du Pont-d'Arve
CH-1211 Geneva 4
Switzerland
0797605012 (Phone)

Bryan T. Kelly

Yale SOM ( email )

135 Prospect Street
P.O. Box 208200
New Haven, CT 06520-8200
United States

AQR Capital Management, LLC ( email )

Greenwich, CT
United States

National Bureau of Economic Research (NBER) ( email )

1050 Massachusetts Avenue
Cambridge, MA 02138
United States

Semyon Malamud (Contact Author)

Ecole Polytechnique Federale de Lausanne ( email )

Lausanne, 1015
Switzerland

Centre for Economic Policy Research (CEPR) ( email )

London
United Kingdom

Swiss Finance Institute

c/o University of Geneva
40, Bd du Pont-d'Arve
CH-1211 Geneva 4
Switzerland

Do you want regular updates from SSRN on Twitter?

Paper statistics

Downloads
279
Abstract Views
667
rank
150,184
PlumX Metrics