Real-Time Sign Language Detection and Interpretation Using Spatiotemporal Deep Learning

Abstract

Sign language plays a significant role in the communication of the hearing and speech impaired. However, the interaction of the hearing and speech impaired with the hearing community of the society has remained a challenge in the absence of proper interpretation mechanisms. It is a challenging task to develop a reliable sign language recognition system in real-time, as the gestures involve complex spatiotemporal information such as the shapes of the hands and the facial expressions. This paper proposes a real-time sign language detection and interpretation system based on spatiotemporal deep learning architectures. A webcam is used to record the gestures, and the video is processed using deep learning architectures. A combination of 3D CNN and attention-based architectures is employed to learn the gestures. The signs are interpreted and translated into text and speech. Experimental results show that the proposed system has high recognition accuracy and operates in real-time. It has been demonstrated that deep learning architectures can be used for the interpretation of gestures and the development of sign language recognition systems.