Towards Explaining Deep Learning: Asymptotic Properties of ReLU FFN Sieve Estimators

42 Pages Posted: 27 Dec 2019 Last revised: 7 Mar 2020

See all articles by Hasan Fallahgoul

Hasan Fallahgoul

Monash University

Vincentius Franstianto

Monash University

Gregoire Loeper

Monash University - School of Mathematical Sciences; Ecole Centrale Paris

Date Written: December 6, 2019

Abstract

A multi-layer, multi-node ReLU network is a powerful, efficient, and popular tool in statistical prediction tasks. However, in contrast to the great emphasis on its empirical applications, its statistical properties are rarely investigated which is mainly due to its severe nonlinearity and heavy parametrization. To help to close this gap via a sieve estimator, we first show that there exists such a sieve estimator for a ReLU feed-forward network. Next, we establish three asymptotic properties of the ReLU network: consistency, sieve-based convergence rate, and asymptotic normality. Finally, to validate the theoretical results, a Monte Carlo analysis is provided.

Keywords: Deep Learning, Neural Networks, Rectified Linear Unit, Sieve Estimators, Consistency, Rate of Convergence

JEL Classification: C1, C5

Suggested Citation

Fallahgoul, Hasan A and Franstianto, Vincentius and Loeper, Gregoire, Towards Explaining Deep Learning: Asymptotic Properties of ReLU FFN Sieve Estimators (December 6, 2019). Available at SSRN: https://ssrn.com/abstract=3499324 or http://dx.doi.org/10.2139/ssrn.3499324

Hasan A Fallahgoul (Contact Author)

Monash University ( email )

Clayton Campus
Victoria, 3800
Australia

HOME PAGE: http://www.hfallahgoul.com

Vincentius Franstianto

Monash University ( email )

23 Innovation Walk
Wellington Road
Clayton, Victoria 3800
Australia

Gregoire Loeper

Monash University - School of Mathematical Sciences ( email )

Clayton Campus
Victoria, 3800
Australia

Ecole Centrale Paris ( email )

Paris
France

Here is the Coronavirus
related research on SSRN

Paper statistics

Downloads
60
Abstract Views
479
rank
374,012
PlumX Metrics