Bidirectional Sign Language Communication Based on Variational and Adversarial Learning

Abstract

Most studies on sign language recognition (SLR) focus on deaf-to-hearing communication, while hearing-to-deaf communication, which can be realized through sign language generation (SLG), has been largely ignored. To address this issue, this paper combines SLR with SLG to construct a skeletonbased bidirectional sign language communication framework. In the SLR stage, a bidirectional long short-term memory (BiLSTM) network is used for the sign language skeleton sequence classifier. In the SLG stage, a novel sign language skeleton sequence generator called SeqαGAN is proposed. Based on variational and adversarial learning, SeqαGAN produces a more flexible posterior distribution and diverse human-recognizable data. The generated samples can be used for data argumentation and improving classifier performance. A series of tests are conducted on three datasets, and the evaluation results indicate that the skeleton-based bidirectional communication framework is effective.

Keywords: SLR, SLG, generation, recognition, bidirectional communication