Dynamic Sparse Coding-Based Value Estimation Network for Deep Reinforcement Learning

Abstract

Deep Reinforcement Learning (DRL) is one powerful tool for varied control automation problems. Performances of DRL highly depend on the accuracy of value estimation for states from environments. However, the Value Estimation Network (VEN) in DRL can be easily influenced by the phenomenon of catastrophic interference from environments and training, and moreover, the efficiency of DRL is restricted by the training with redundant parameters. In this paper, we propose a Dynamic Sparse Coding-based (DSC) DRL model to obtain precise sparse representations and sparse parameters in VEN through training, i.e., DSC-VEN, to improve the control performance, which is not only applicable in Q-learning but also in actor-critic structured DRL. In detail, to alleviate interference in VEN, we propose to employ dynamic sparse coding to learn sparse representations for accurate value estimation with a dynamic sparsity regularizer. To avoid influences from redundant parameters, we employ dynamic sparse coding to prune weights with dynamic thresholds. Therefore, we develop a dynamic sparse coding-based VEN algorithm, which can improve control performances not only for Q-learning structured discrete-action DRL but also actor-critic structured continuous-action DRL. We have validated the proposed DSC-VEN in both discrete and continuous action benchmark environments. Experiments demonstrate that the proposed algorithms with dynamic sparse coding can obtain higher control performances than existing benchmark DRL algorithms in both discrete-action and continuous-action environments, e.g., over 25% increase in Puddle World and about 10% increase in Hopper. Moreover, the proposed algorithm can reach convergence efficiently with fewer episodes in different environments.

Keywords: deep reinforcement learning, value estimation network, dynamic sparse coding