Real-Time Anomaly Detection for Distributed Systems Logs Using Apache Kafka and H2O.ai - Daugėla, Kęstutis; Vaičiukynas, Evaldas (Information and Software Technologies, 2022)
System monitoring is crucial to ensure that the system is working correctly. Usually, it encompasses solutions from the simple configuration of static thresholds for hardware/software key performance indicators to employing anomaly detection algorithms on a stream of numerical data. System logs, on the other hand, is another golden source of the system state, but often it is overlooked. Combining system logs with load metrics could potentially increase the accuracy of anomaly detection. We propose a robust pipeline and evaluate several of its variants for solving such a task at scale and in real-time. Experiments with proprietary logs from an enterprise Kafka cluster reveal that pre-processing with an autoencoder prior to applying the isolation forest method can significantly improve the detection performance.