A New Likelihood Ratio Method for Training Artificial Neural Networks

33 Pages Posted: 7 Feb 2019 Last revised: 11 May 2021

See all articles by Yijie Peng

Yijie Peng

Peking University

Li Xiao

Chinese Academy of Sciences (CAS)

Bernd Heidergott

VU University Amsterdam

L. Jeff Hong

Hong Kong University of Science & Technology (HKUST)

Henry Lam

Columbia University

Date Written: January 19, 2019

Abstract

We investigate a new approach to compute the gradients of artificial neural networks (ANNs), based on the so-called push-out likelihood ratio method. Unlike the widely used backpropagation (BP) method that requires continuity of the loss function and the activation function, our approach bypasses this requirement by injecting artificial noises into the signals passed along the neurons. We show how this approach has a similar computational complexity as BP, and moreover is more advantageous in terms of removing the backward recursion and eliciting transparent formulas. We also formalize the connection between BP, a pivotal technique for training ANNs, and infinitesimal perturbation analysis, a classic path-wise derivative estimation approach, so that both our new proposed methods and BP can be better understood in the context of stochastic gradient estimation. Our approach allows efficient training for ANNs with more flexibility on the loss and activation functions, and shows empirical improvements on the robustness of ANNs under adversarial attacks and corruptions of natural noises.

Keywords: simulation, stochastic gradient estimation, artificial neural network, image identification

JEL Classification: C18

Suggested Citation

Peng, Yijie and Xiao, Li and Heidergott, Bernd and Hong, L. Jeff and Lam, Henry, A New Likelihood Ratio Method for Training Artificial Neural Networks (January 19, 2019). Available at SSRN: https://ssrn.com/abstract=3318847 or http://dx.doi.org/10.2139/ssrn.3318847

Yijie Peng

Peking University ( email )

No 5 Yiheyuan Rd
Haidian District
Beijing, Beijing 100871
China

Li Xiao (Contact Author)

Chinese Academy of Sciences (CAS) ( email )

52 Sanlihe Rd.
Datun Road, Anwai
Beijing, Xicheng District 100864
China

Bernd Heidergott

VU University Amsterdam ( email )

Netherlands

L. Jeff Hong

Hong Kong University of Science & Technology (HKUST) ( email )

Department of Industrial Engineering and Logistics
Hong Kong University of Science and Technology
Hong Kong, 00000 00000
Hong Kong

Henry Lam

Columbia University ( email )

3022 Broadway
New York, NY 10027
United States

Do you have a job opening that you would like to promote on SSRN?

Paper statistics

Downloads
194
Abstract Views
1,054
Rank
239,849
PlumX Metrics