SwissLog: Robust Anomaly Detection and Localization for Interleaved Unstructured Logs

• Published in: IEEE transactions on dependable and secure computing Vol. 20; no. 4; pp. 2762 - 2780
• Main Authors: Li, Xiaoyun, Chen, Pengfei, Jing, Linxiao, He, Zilong, Yu, Guangba
• Format: Journal Article
• Language: English
• Published: Washington IEEE 01.07.2023; IEEE Computer Society
• Subjects: Anomalies; Anomaly detection; anomaly localization; Codes; Computer networks; Containers; Cybersecurity; Deep learning; Localization; Location awareness; log correlation; log parsing; Messages; Parsers; Robustness; Search algorithms; Semantics; Software systems; Synthetic data; Tuning
• ISSN: 1545-5971; 1941-0018
• DOI: 10.1109/TDSC.2022.3162857

Modern distributed systems generate interleaved logs when running in parallel. Identifiers (ID) are always attached to them to trace running instances or entities in logs. Therefore, log messages can be grouped by the same IDs to help anomaly detection and localization. The existing approaches to achieve this still fall short meeting these challenges: 1) Log is solely processed in single components without mining log dependencies. 2) Log formats are continually changing in modern software systems. 3) It is challenging to detect latent performance issues non-intrusively by trivial monitoring tools. To remedy the above shortcomings, we propose SwissLog, a robust anomaly detection and localization tool for interleaved unstructured logs. SwissLog focuses on log sequential anomalies and tries to dig out possible performance issues. SwissLog constructs ID relation graphs across distributed components and groups log messages by IDs. Moreover, we propose an online data-driven log parser without parameter tuning. The grouped log messages are parsed via the novel log parser and transformed with semantic and temporal embedding. Finally, SwissLog utilizes an attention-based Bi-LSTM model and a heuristic searching algorithm to detect and localize anomalies in instance-granularity, respectively. The experiments on real-world and synthetic datasets confirm the effectiveness, efficiency, and robustness of SwissLog.