AI-Powered Multimodal Interviewer: An Integrated Framework for Automated Candidate Evaluation Using Speech Recognition, NLP, and Facial Emotion Analysis

Abstract

Recruitment processes are subjectively influenced when done manually and inefficiently compared to automated processes. This project has developed an intelligent automated interviewer based on artificial intelligence (AI) which combines speech recognition technology, natural language processing (NLP), facial emotion recognition/analysis and voice verification in order to create a holistic evaluation of candidates. The system allows for real-time evaluation of technical skills, communication skills and behavior, in addition to storing text, audio and video as three independent modalities in one interview session. The transformer model called BERT evaluates candidate response relevance and coherence, while emotion detection is performed using computer vision (OpenCV and MediaPipe) and voice biometrics provide an authenticated identity of the candidate to avoid impersonation. Anti-cheating mechanisms check for interview integrity using computer vision. All session data (analytical reports), emotional metrics and automated feedback will be stored centrally in a MongoDB database for administrative review. Experimental results confirm that this system has been successfully deployed with real-time multimodal fusion and consistent evaluation. The proposed AI-powered interviewer is available to the employment market as an unbiased, scalable and data-driven solution for evaluating candidates in recruitment, training and academic environments.