Towards Explaining Deep Learning: Asymptotic Properties of ReLU FFN Sieve Estimators
62 Pages Posted: 27 Dec 2019 Last revised: 6 Sep 2022
Date Written: December 6, 2019
Abstract
Recently, machine learning algorithms have increasing become popular tools for economic and financial forecasting. While there are several machine learning algo- rithms for doing so, a powerful and efficient algorithm for forecasting purposes is the multi-layer, multi-node neural network with rectified linear unit (ReLU) activa- tion function – deep neural network (DNN). Studies have demonstrated the empir- ical applications of DNN but have devoted less research to investigate its statistical properties which is mainly due to its severe nonlinearity and heavy parametrization. By borrowing tools from a non-parametric regression framework, sieve estimator, we first show that there exists such a sieve estimator for a DNN. We next establish three asymptotic properties of the ReLU network: consistency, sieve-based convergence rate, and asymptotic normality, and then validate our theoretical results using Monte Carlo analysis.
Keywords: Deep Learning, Neural Networks, Rectified Linear Unit, Sieve Estimators, Consistency, Rate of Convergence
JEL Classification: C1, C5
Suggested Citation: Suggested Citation