Data-driven monitoring for cloud compute systems

• Published in: 2016 IEEE ACM 9th International Conference on Utility and Cloud Computing (UCC) pp. 128 - 137
• Main Authors: Gehberger, Daniel, Matray, Peter, Nemeth, Gabor
• Format: Conference Proceeding
• Language: English
• Published: New York, NY, USA ACM 06.12.2016
• Series: ACM Other Conferences
• Subjects: Cloud computing; Computational modeling; Industrial Internet of Things; Logic gates; Measurement; Monitoring; Optimization; Real-time systems; Topology
• ISBN: 1450346162; 9781450346160
• DOI: 10.1145/2996890.2996893

The end-to-end monitoring of inter-dependent applications in the cloud is challenging. Difficulties arise from the complexity of computations and the highly distributed nature of the deployment. Due to the lack of a comprehensive observability solution, it is very difficult to apply autonomous mechanisms to ensure service guarantees in the cloud. To tackle the problem, we propose the method of data-driven monitoring, that provides a detailed, live view on how data is flowing through a possibly complex compute system. The method is based on the tracing of individual input events and the collection of resource usage metrics along the paths. By reconstructing causal and temporal relationships, we can detect degradations in performance, pinpoint root causes and apply corrective actions before end-to-end requirements are endangered. To demonstrate the potential of the concept, we created a prototype implementation in a big data compute platform, and also developed two automated optimization algorithms.