Towards Explaining the ReLU Feed-Forward Network
56 Pages Posted: 27 Dec 2019 Last revised: 28 Dec 2019
Date Written: December 6, 2019
A multi-layer, multi-node ReLU network is a powerful, efficient, and popular tool in statistical prediction tasks. However, in contrast to the great emphasis on its empirical applications, its statistical properties are rarely investigated. To help closing this gap, we establish three asymptotic properties of the ReLU network: consistency, sieve-based convergence rate, and asymptotic normality. To validate the theoretical results, a Monte Carlo analysis is provided.
Keywords: Consistency, Rate of Convergence, Sieve Estimators, Rectified Linear Unit
JEL Classification: G1, C1
Suggested Citation: Suggested Citation