Deep Regression Ensembles

29 Pages Posted: 8 Mar 2022

See all articles by Antoine Didisheim

Antoine Didisheim

The University of Melbourne; Swiss Finance Institute

Bryan T. Kelly

Yale SOM; AQR Capital Management, LLC; National Bureau of Economic Research (NBER)

Semyon Malamud

Ecole Polytechnique Federale de Lausanne; Centre for Economic Policy Research (CEPR); Swiss Finance Institute

Date Written: March 3, 2022

Abstract

We introduce a methodology for designing and training deep neural networks (DNN) that we call 'Deep Regression Ensembles' (DRE). It bridges the gap between DNN and two-layer neural networks trained with random feature regression. Each layer of DRE has two components, randomly drawn input weights and output weights trained myopically (as if the final output layer) using linear ridge regression. Within a layer, each neuron uses a different subset of inputs and a different ridge penalty, constituting an ensemble of random feature ridge regressions. Our experiments show that a single DRE architecture is at par with or exceeds state-of-the-art DNN in many data sets. Yet, because DRE neural weights are either known in closed-form or randomly drawn, its computational cost is orders of magnitude smaller than DNN.

Keywords: Deep learning, Neural network, Random features, Ensembles

Suggested Citation

Didisheim, Antoine and Kelly, Bryan T. and Malamud, Semyon, Deep Regression Ensembles (March 3, 2022). Swiss Finance Institute Research Paper No. 22-20, 2022, Available at SSRN: https://ssrn.com/abstract=4049493 or http://dx.doi.org/10.2139/ssrn.4049493

Antoine Didisheim

The University of Melbourne ( email )

Parkville, 3010
Australia
0435776821 (Phone)

Swiss Finance Institute ( email )

University of Melbourne
Melbourne, VA
Australia
0797605012 (Phone)

Bryan T. Kelly

Yale SOM ( email )

135 Prospect Street
P.O. Box 208200
New Haven, CT 06520-8200
United States

AQR Capital Management, LLC ( email )

Greenwich, CT
United States

National Bureau of Economic Research (NBER) ( email )

1050 Massachusetts Avenue
Cambridge, MA 02138
United States

Semyon Malamud (Contact Author)

Ecole Polytechnique Federale de Lausanne ( email )

Lausanne, 1015
Switzerland

Centre for Economic Policy Research (CEPR) ( email )

London
United Kingdom

Swiss Finance Institute

c/o University of Geneva
40, Bd du Pont-d'Arve
CH-1211 Geneva 4
Switzerland

Do you have negative results from your research you’d like to share?

Paper statistics

Downloads
932
Abstract Views
2,613
Rank
46,705
PlumX Metrics