Real-Time Big Data Analytics with Apache Flink and Python APIs

Abstract

Real-time big data analytics plays an important role in modern Internet of Things environments where large volumes of sensor data are generated continuously. Traditional batch processing systems analyze data only after storage, which causes delays in detecting critical events. This paper presents a real-time big data analytics system using Apache Flink and Python APIs for processing IoT sensor streams. The proposed system simulates sensor readings such as temperature, humidity, pressure, and vibration from multiple IoT devices. The generated data is processed using a real-time stream processing pipeline that performs anomaly detection, window-based aggregation, statistical analysis, visualization, and report generation. Threshold-based anomaly detection is used to identify abnormal sensor behavior instantly. The system also applies tumbling, sliding, and session window concepts to summarize continuous data streams over time. Experimental evaluation was carried out using 600 simulated sensor readings from five sensors. The system detected 45 anomalous readings with an anomaly rate of 7.50 percent and achieved a throughput of 3.34 records per second in the simulated environment. The results show that Apache Flink with Python APIs can be used effectively for low-latency real-time analytics, IoT monitoring, and anomaly detection applications.