Bridging Communication Gaps: A CNN-RNN Powered System for Real-Time Indian Sign Language Recognition and Full Sentence Translation

Abstract

Sign language is essential for people who are deaf or have speech difficulties, yet the shortage of interpreters often leaves them isolated in education, work, and social life. This study introduces a fast, reliable deep-learning system for recognizing and translating Indian Sign Language (ISL) in real time. Using Google Mediapipe, we extract stable three-dimensional hand landmarks from video frames. The resulting sequences of keypoints are fed into a hybrid model that combines a Convolutional Neural Network (CNN) for spatial feature extraction with a Gated Recurrent Unit (GRU) for temporal dynamics. Our custom dataset contains 3000+ video clips of 65 ISL sentences spoken by 11 different signers. The network, with only 1.5 million parameters, reaches 93.64% accuracy while maintaining a low computational cost and real-time inference speed.