Towards Explaining Deep Learning: Asymptotic Properties of ReLU FFN Sieve Estimators
42 Pages Posted: 27 Dec 2019 Last revised: 7 Mar 2020
Date Written: December 6, 2019
A multi-layer, multi-node ReLU network is a powerful, efficient, and popular tool in statistical prediction tasks. However, in contrast to the great emphasis on its empirical applications, its statistical properties are rarely investigated which is mainly due to its severe nonlinearity and heavy parametrization. To help to close this gap via a sieve estimator, we first show that there exists such a sieve estimator for a ReLU feed-forward network. Next, we establish three asymptotic properties of the ReLU network: consistency, sieve-based convergence rate, and asymptotic normality. Finally, to validate the theoretical results, a Monte Carlo analysis is provided.
Keywords: Deep Learning, Neural Networks, Rectified Linear Unit, Sieve Estimators, Consistency, Rate of Convergence
JEL Classification: C1, C5
Suggested Citation: Suggested Citation