Design of Enterprise Economic Dynamic Management System Based on Spark Technology

• Published in: Multimedia Technology and Enhanced Learning Vol. 446; pp. 274 - 286
• Main Authors: Zhang, Lu, Yan, Yipin
• Format: Book Chapter
• Language: English
• Published: Switzerland Springer 2022; Springer Nature Switzerland
• Series: Lecture Notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering
• Subjects: Data warehouse; Dynamic management; Enterprise economy; Spark Join operator; Spark technology
• ISBN: 3031181220; 9783031181221
• ISSN: 1867-8211; 1867-822X
• DOI: 10.1007/978-3-031-18123-8_21

Aiming at the problem that the currently used dynamic management system based on Hadoop and B/S architecture is affected by the slow data mining rate, resulting in low efficiency of data dynamic management, a design of enterprise economic dynamic management system based on Spark technology is proposed. Deploy the physical architecture of the Spark-based economic dynamic management system, and build a data warehouse in this architecture to facilitate users to quickly view data in real time. The B/S (browser/server) model is adopted to design data collection modules, business service modules and performance modules to meet the needs of big data analysis and decision-making. When using Spark technology to dynamically adjust the difference data in the database, the rule base needs to be updated in time to convert the automatic conversion system to a detection system. Use the optimization algorithm of Spark Join operator to optimize the entire connection operation, filter out the project data without specific categories in the bank flow data, reduce the data entering the shuffle stage, and design a dynamic management process. It can be seen from the test results that the system has a maximum management efficiency of 92% in a safe environment. In a non-interference environment, the highest management efficiency is 0.95, which has an efficient management effect.