Emotion Recognition for Human Behavior Analysis Using AI & IoT Systems

Abstract

The analysis of human behavior and states of mind through first-person experiences is always a challenging issue due to problems with portability, privacy, and clarity associated with existing solutions. In this paper, we propose a new IoT-based system that integrates AI capabilities with a spectacles-based XAIO ESP32-S3 Sense module to obtain synchronized audio and video signals. The Python-based modular backend is utilized to preprocess both signals, recognize facial landmarks using Insight Face, recognize speech emotions using Speech Brain and Whisper, and identify behavioral factors. The Gemini 2.5 Flash-based reporting engine is then utilized to create dynamic, intention-based reports with layered narratives, behavioral factor analysis, and export options in various formats, such as PDF, JSON, TXT, and CSV. The entire system is designed to operate within a local framework to ensure maximum privacy. The proposed system is applicable to various scenarios, such as interviews, meetings, and counseling, as it is capable of generating human-readable psychological reports without depending on any external database. The paper presents a complete end-to-end wearable behavior analysis system.