Fast Outlier Detection Using a Grid-Based Algorithm

• Published in: PloS one Vol. 11; no. 11; p. e0165972
• Main Authors: Lee, Jihwan, Cho, Nam-Wook
• Format: Journal Article
• Language: English
• Published: United States Public Library of Science 10.11.2016; Public Library of Science (PLoS)
• Subjects: Algorithms; Artificial intelligence; Cluster Analysis; Computation; Computer and Information Sciences; Computer applications; Computer security; Computer Systems - economics; Cybersecurity; Data analysis; Data mining; Data Mining - economics; Data Mining - methods; Data processing; Datasets; Energy research; International conferences; Linear algebra; Mathematical analysis; Methods; Normal distribution; Nuclear electric power generation; Nuclear energy; Outliers (statistics); People and Places; Physical Sciences; Research and Analysis Methods; Time Factors; New York; United States > US; South Korea
• ISSN: 1932-6203; 1932-6203
• DOI: 10.1371/journal.pone.0165972

As one of data mining techniques, outlier detection aims to discover outlying observations that deviate substantially from the reminder of the data. Recently, the Local Outlier Factor (LOF) algorithm has been successfully applied to outlier detection. However, due to the computational complexity of the LOF algorithm, its application to large data with high dimension has been limited. The aim of this paper is to propose grid-based algorithm that reduces the computation time required by the LOF algorithm to determine the k-nearest neighbors. The algorithm divides the data spaces in to a smaller number of regions, called as a "grid", and calculates the LOF value of each grid. To examine the effectiveness of the proposed method, several experiments incorporating different parameters were conducted. The proposed method demonstrated a significant computation time reduction with predictable and acceptable trade-off errors. Then, the proposed methodology was successfully applied to real database transaction logs of Korea Atomic Energy Research Institute. As a result, we show that for a very large dataset, the grid-LOF can be considered as an acceptable approximation for the original LOF. Moreover, it can also be effectively used for real-time outlier detection.