Multilingual Text-Based Augmentative Communication Using Eye Tracking and Predictive Language Models

Abstract

Effective human-computer interaction remains a major challenge for people with severe motor impairments since most of them have very limited access to conventional input devices like keyboards and mice. Augmentative and Alternative Communication systems address the challenge; however, many of the existing solutions are expensive, complicated, and linguistically restrictive. The present work proposes a low-cost, text-based AAC platform that effectively integrates webcam-based eye and head movement tracking with intelligent language prediction to enhance communicational efficiency. The system utilizes techniques from computer vision for estimating gaze direction and head pose and allows for virtual keyboard-based text entry without using any special hardware. In this work, to further improve the typing speed and reduce input errors, a neural language model generates context-aware word suggestions during the text composition process. Unlike most existing AAC solutions that are bound to a single language, the proposed framework supports multilingual communication by translating the generated text into the user’s preferred language, hence extending the accessibility for diverse linguistic populations. The system’s performance is evaluated in terms of typing performance, prediction accuracy, and adaptability; it stands out as an effective, low-resource, and inclusive communication solution for users with significant motor disabilities.