Towards a Cloud Native Big Data Platform using MiCADO

• Published in: 2020 19th International Symposium on Parallel and Distributed Computing (ISPDC) pp. 118 - 125
• Main Authors: Mosa, Abdelkhalik, Kiss, Tamas, Pierantoni, Gabriele, DesLauriers, James, Kagialis, Dimitrios, Terstyanszky, Gabor
• Format: Conference Proceeding
• Language: English
• Published: IEEE 01.07.2020
• Subjects: Big Data; Cloud computing; cloud native; Container-orchestration; Containers; Docker Containers; Hadoop big data platform; MiCADO; Monitoring; scaling big data infrastructure; Software; Task analysis
• DOI: 10.1109/ISPDC51135.2020.00025

In the big data era, creating self-managing scalable platforms for running big data applications is a fundamental task. Such self-managing and self-healing platforms involve a proper reaction to hardware (e. g., cluster nodes) and software (e. g., big data tools) failures, besides a dynamic resizing of the allocated resources based on overload and underload situations and scaling policies. The distributed and stateful nature of big data platforms (e. g., Hadoop-based cluster) makes the management of these platforms a challenging task. This paper aims to design and implement a scalable cloud native Hadoop-based big data platform using MiCADO, an open-source, and a highly customisable multi-cloud orchestration and auto-scaling framework for Docker containers, orchestrated by Kubernetes. The proposed MiCADO-based big data platform automates the deployment and enables an automatic horizontal scaling (in and out) of the underlying cloud infrastructure. The empirical evaluation of the MiCADO-based big data platform demonstrates how easy, efficient, and fast it is to deploy and undeploy Hadoop clusters of different sizes. Additionally, it shows how the platform can automatically be scaled based on user-defined policies (such as CPU-based scaling).