Real-Time Anomaly Detection for Distributed Systems Logs Using Apache Kafka and H2O.ai

• Published in: Information and Software Technologies Vol. 1665; pp. 33 - 42
• Main Authors: Daugėla, Kęstutis, Vaičiukynas, Evaldas
• Format: Book Chapter
• Language: English
• Published: Switzerland Springer International Publishing AG 2022; Springer International Publishing
• Series: Communications in Computer and Information Science
• Subjects: Anomaly detection; Apache Kafka; Autoencoder; H2O.ai; Isolation forest; System logs
• ISBN: 9783031163012; 303116301X
• ISSN: 1865-0929; 1865-0937
• DOI: 10.1007/978-3-031-16302-9_3

System monitoring is crucial to ensure that the system is working correctly. Usually, it encompasses solutions from the simple configuration of static thresholds for hardware/software key performance indicators to employing anomaly detection algorithms on a stream of numerical data. System logs, on the other hand, is another golden source of the system state, but often it is overlooked. Combining system logs with load metrics could potentially increase the accuracy of anomaly detection. We propose a robust pipeline and evaluate several of its variants for solving such a task at scale and in real-time. Experiments with proprietary logs from an enterprise Kafka cluster reveal that pre-processing with an autoencoder prior to applying the isolation forest method can significantly improve the detection performance.