AI Based Transaction Anomaly and Fraud Detection System

Abstract

The fast growth of digital payment systems has led to a substantial rise in both the volume and complexity of fraudulent financial transactions.  Traditional rule-based fraud detection methods lack flexibility and often fail to recognize new or previously unknown fraud patterns.This paper presents a hybrid explainable artificial intelligence (XAI) framework for real-time financial fraud detection. The proposed approach combines supervised learning using the XGBoost algorithm with unsupervised anomaly detection through Isolation Forest, enabling the system to identify both known fraud types and emerging suspicious activities.To enhance detection performance, the framework incorporates behavioral profiling, temporal transaction analysis, merchant risk evaluation, and device-level consistency features. Additionally, explainable AI techniques based on SHAP are applied to generate clear and interpretable explanations for each fraud prediction.Experimental results show that the hybrid model achieves higher recall and a lower false-positive rate compared to individual models, demonstrating its effectiveness and suitability for deployment in modern banking systems and digital payment platforms.