Real-Time Gesture Recognition for Sign Language Using Multimodal Techniques

Abstract

Sign language serves as a crucial communication medium for individuals with hearing and speech impairments. However, existing recognition systems primarily focus on isolated word detection, limiting their effectiveness for real-time communication. This paper presents a Continuous Sign Language Recognition System using a multimodal approach to seamlessly interpret sign language gestures into coherent words. The system leverages Mediapipe, a robust computer vision framework, to detect and track hand gestures, body posture, and facial landmarks in real time. By processing keypoints extracted from these modalities, the system enhances recognition accuracy and ensures fluid word construction. Unlike traditional methods that rely on gloves or additional hardware, this approach offers a cost-effective and accessible solution for real-time sign language translation. Experimental evaluations demonstrate the system’s ability to accurately recognize continuous gestures, addressing challenges such as real-time performance, word-level interpretation, and scalability. The proposed model significantly improves accessibility and inclusivity, bridging the communication gap between sign language users and non-signers.