Arabic Sign Language Recognition using Microsoft Kinect and Leap Motion Controller

Abstract

Communication is an essential and very important throughout everyday life, sign language is an expressive way which enables the deaf people to interact with their societies. Several researches were introduced in sign language recognition systems but till now developing an accurate realtime system for sign language recognition is still a challenge and applications in (ArSL) Arabic sign language is very limited. This paper proposes Arabic hand gestures recognition model based on using two depth sensors namely Microsoft Kinect and Leap Motion controller. The proposed model uses two direct matching learning algorithms: a) Dynamic Time Wrapping (DTW) and, b) Hidden Markov Model (HMM) then applies DSmT (Dezert-Smarandache Theory) to recognize the captured gestures. The model is applied on collected data set of 30 hand gestures which are composed of 20 single-hand gestures and 10 double-hand gestures. The overall accuracy of the single-hand model is 89.9% for HMM, 91.05% for DTW and 93.35% after applying DSmT fusion. The overall accuracy of the double-hand model is 86.3%, 88.1% and 91% for HMM, DTW and DSmT fusion respectively.

Keywords: Arabic sign language, Dynamic Time Wrapping, Hidden Markov Model, Dezert-Smarandache Theory