Efficient Self-Learning Evolutionary Neural Architecture Search

Abstract

The evolutionary algorithm has become a major method for neural architecture search recently. However, the fixed probability distribution adopted by the traditional evolutionary algorithm makes the algorithm unable to control the increase or decrease of the individual architecture size and thus cannot guarantee the lightweight and inference efficiency of the candidate architecture. In addition, the existing approaches cannot learn the optimal sampling probability distribution relevant to the specific problem from the empirical information accumulated through the search process. What’s more, the algorithm needs to evaluate the performance of all the individual architectures, which requires huge computing resources and time overhead. To overcome these problems and challenges, an Efficient Self-learning Evolutionary Neural Architecture Search method, called ESE-NAS, is presented in this paper. Firstly, we propose a Model Size Control module, which generates the probability distribution for sampling the type of mutation operators according to the current model size, so as to control both the node number of network architecture and the sparsity of links, therefore ensuring that neural architectures can remain compact and efficient as they evolve. Thereafter, a Mutation Candidate Credit Assignment method is proposed, which enables the algorithm to dynamically adjust the probability distribution for available types of node operations and directed links according to the empirical information from individuals’ performance evaluation, which can guide the evolutionary direction of neural architecture and shorten the first hitting time of the optimal architecture. Finally, a neural architecture performance predictor is formulated, and the accuracy of each individual architecture is predicted by using the simplified input features, to further improve the efficiency of neural architecture search. Experiments show that the ESE-NAS method proposed in this paper can learn a highly interpretable credit assignment results and can significantly bring forward the first hitting time of optimal architecture compared with the method using a fixed probability distribution. The performances of the searched neural architectures are quite competitive with the SOTA representative hand-designed and NAS architectures, while the proposed ESE-NAS effectively guarantee the simplicity and inference efficiency of the neural architectures at the same time.

Keywords: evolutionary algorithm, neural architecture search, Probability Distribution, Model Size Control