Using PyArrow for Efficient Data Serialization and Transfer in Big Data Systems

Abstract

Efficient data serialization and transfer are essential in big data systems, where large volumes of structured data must be stored, processed, and transmitted with minimal overhead. Traditional formats such as CSV, JSON, and Pickle are widely used for data storage and exchange, but they often have limitations in terms of serialization speed, deserialization speed, file size, memory efficiency, throughput, and latency. This project focuses on addressing these challenges by developing a web-based platform that demonstrates and compares efficient data serialization techniques using PyArrow. The main objective of this project is to build a Flask-based big data serialization platform that enables users to upload datasets, preview data, analyze schema information, convert datasets into multiple formats, and evaluate performance through benchmark metrics. The platform supports multiple structured data formats including CSV, JSON, Excel, Pickle, Arrow IPC, Feather, and Parquet. After uploading a dataset, the system processes the data using Pandas for validation, preview, and conversion into a PyArrow Table. It also includes advanced features such as Parquet compression comparison, memory mapping, dictionary encoding, metadata extraction, and report generation. The results demonstrate the effectiveness of PyArrow in improving data storage and transfer efficiency in big data applications. Different compression techniques show varying performance in terms of speed, storage efficiency, and transfer performance. Some methods provide better compression with smaller file sizes, while others offer faster serialization and lower latency. Overall, the developed system serves as a practical tool for comparing serialization formats and helps users choose the most suitable format based on performance, storage requirements, and data transfer efficiency.