Towards an Energy Complexity Model for Distributed Data Processing Algorithms

• Published in: IEEE transactions on big data Vol. 9; no. 6; pp. 1 - 13
• Main Authors: Song, Jie, Zhao, Xingchen, Guo, Chaopeng, Gu, Yu, Yu, Ge
• Format: Journal Article
• Language: English
• Published: Piscataway IEEE 01.12.2023; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Algorithms; Big Data; Complexity; Complexity theory; Computational modeling; Computer centers; Data centers; Data processing; Distributed data processing algorithm; Energy complexity; Energy consumption; Energy costs; Green computing; Hardware; Optimization; Servers; Software; Software algorithms
• ISSN: 2332-7790; 2372-2096
• DOI: 10.1109/TBDATA.2023.3284259

Modern data centers exist as infrastructure in the era of big data. Big data processing applications are the major computing workload of data centers. Electricity cost accounts for about 50% of data centers' operational costs. Therefore, the energy consumed for running distributed data processing algorithms on a data center is starting to attract both academia and industry. Most works study the energy consumption from the hardware perspective and only a few of them from the algorithm perspective. A general and hardware-independent energy evaluation model for the algorithms is in demand. With the model, algorithm designers can evaluate the energy consumption, compare energy consumption features and facilitate energy consumption optimization of distributed data processing algorithms. Inspired by the time complexity model, we propose an energy complexity model for describing the trends that an algorithm's energy consumption grows with the algorithm's input size. We argue that a good algorithm, especially for processing big data, should have a 'small' energy complexity. We define <inline-formula><tex-math notation="LaTeX">E(n)</tex-math></inline-formula> to represent the functional relationship that associates an algorithm's input size <inline-formula><tex-math notation="LaTeX">n</tex-math></inline-formula> with its notional energy consumption <inline-formula><tex-math notation="LaTeX">E</tex-math></inline-formula>. Based on the well-known abstract Bulk Synchronous Parallel (BSP) computer and programming model, we present a complete <inline-formula><tex-math notation="LaTeX">E(n)</tex-math></inline-formula> solution, including abstraction, generalization, quantification, derivation, comparison, analysis, examples, verification, and applications. Comprehensive experimental analysis shows that the proposed energy complexity model is practical, interestingly, and not equivalent to time complexity.