Statistically Based Weight Pruning in Feed-Forward Neural Networks

41 Pages Posted: 10 Mar 2012

See all articles by Richard Briesch

Richard Briesch

Southern Methodist University (SMU), Edwin L. Cox School of Business, Marketing Department (Deceased)

Date Written: March 9, 2012

Abstract

A statistically-based algorithm for pruning weights from feed-forward networks is presented. This algorithm relies upon the Generalized Wald and t-test statistics to determine which weights to remove from the network. Because both of these tests use the exact Hessian matrix, an algorithm for learning the exact Hessian matrix for a feed-forward neural network using a single backward pass through the data is presented when the L2 norm is minimized in the energy function. The pruning algorithm is then applied in two simulations: The first simulation investigates the relationship between neural networks and linear regression (Ordinary Least Squares), and the weight covariance matrix is found to be asymptotically equivalent to the White (1980) standard error corrections for heterogeneity of variance. The final simulation applies the algorithm to a network solving the sunspot data and compares the results to those found in the literature, with mixed results.

Keywords: neural networks, t-test statistics, linear regression, white standard errors, Hessian, model selection, Wald statistic

Suggested Citation

Briesch, Richard, Statistically Based Weight Pruning in Feed-Forward Neural Networks (March 9, 2012). Available at SSRN: https://ssrn.com/abstract=2019120 or http://dx.doi.org/10.2139/ssrn.2019120

Richard Briesch (Contact Author)

Southern Methodist University (SMU), Edwin L. Cox School of Business, Marketing Department (Deceased) ( email )

United States

Do you have a job opening that you would like to promote on SSRN?

Paper statistics

Downloads
94
Abstract Views
1,100
Rank
527,646
PlumX Metrics