Big data processing using hybrid Gaussian mixture model with salp swarm algorithm

• Published in: Journal of big data Vol. 11; no. 1; pp. 167 - 29
• Main Authors: Saravanakumar, R., TamilSelvi, T., Pandey, Digvijay, Pandey, Binay Kumar, Mahajan, Darshan A., Lelisho, Mesfin Esayas
• Format: Journal Article
• Language: English
• Published: Cham Springer International Publishing 01.12.2024; Springer Nature B.V; SpringerOpen
• Subjects: Algorithms; Big Data; Clustering; Communications Engineering; Computational Science and Engineering; Computer Science; Data Mining and Knowledge Discovery; Data processing; Database Management; Gaussian mixture model (GMM); Gaussian process; Hadoop distributed file system (HDFS); Information Storage and Retrieval; Map-reduce; Mathematical Applications in Computer Science; Networks; Optimization; Optimization techniques; Parameter sensitivity; Probabilistic models; Salp swarm algorithm (SSA); Secure hash algorithms (SHA); Secure hash algorithms (SHA); Gaussian mixture model (GMM); Hadoop distributed file system (HDFS); Salp swarm algorithm (SSA); Map-reduce
• ISSN: 2196-1115; 2196-1115
• DOI: 10.1186/s40537-024-01015-3

The traditional methods used in big data, like cluster creation and query-based data extraction, fail to yield accurate results on massive networks. To address such issues, the proposed approach involves using the Hadoop Distributed File System (HDFS) for data processing, the map-reduce programming paradigm for data processing, and query optimization techniques to quickly and effectively extract accurate outcomes from a variety of options with a high processing capacity. The methodology proposed in this work makes use of Gaussian Mixture Model (GMM) for data clustering and the Salp Swarm Algorithm (SSA) for optimization. The security of preprocessed data stored on networked clusters with interconnections has been ensured by SHA algorithms. Finally, incorporating into consideration the important parameters, evaluation findings for the experimental performance of the model in the indicated methodology are produced. For this work, the estimated range of input file sizes is 60–100 MB. The processing of 100 MB of input files yielded an accuracy of 96% and results for specificity and sensitivity of 90% and 93%, respectively. The outcomes have been compared with well-known methods like fuzzy C-means and K-means approaches, and the results show that the proposed method effectively distributes accurate data processing to cluster nodes with low latency. Moreover, it uses the least amount of memory resources possible when operating on functional CPUs. As a result, the proposed approach outperforms existing techniques.