Multilayer Feedforward Networks with a Non-Polynomial Activation Function Can Approximate Any Function

Abstract

Several researchers characterized the activation function under which multilayer feedforwardnetworks can act as universal approximators. We show that most of all the characterizationsthat were reported thus far in the literature are special cases of the followinggeneral result: a standard multilayer feedforward network with a locally bounded piecewisecontinuous activation function can approximate any continuous function to any degree ofaccuracy if and only if the network's activation function is not a polynomial. We alsoemphasize the important role of the threshold, asserting that without it the last theoremdoes not hold.

Keywords: Multilayer feedforward networks, Activation functions, role of threshold, Universal approximation capabilities, LP(μ) approximation