Detecting anomalous patterns in time-series data using sparse hierarchically parameterized transition matrices

• Published in: Pattern analysis and applications : PAA Vol. 20; no. 4; pp. 1029 - 1043
• Main Authors: Milo, Michael W., Roan, Michael J.
• Format: Journal Article
• Language: English
• Published: London Springer London 01.11.2017; Springer Nature B.V
• Subjects: Accuracy; Computer Science; Computing time; Data analysis; Economic models; Markov chains; Pattern Recognition; Quality assurance; Random variables; Robustness (mathematics); Search algorithms; Theoretical Advances; Time series; Statistical models; Monte Carlo; Stochastic processes; Machine learning; Markov processes; Signal processing; Real-time systems; Pattern recognition
• ISSN: 1433-7541; 1433-755X
• DOI: 10.1007/s10044-016-0544-0

Anomaly detection in time-series data is a relevant problem in many fields such as stochastic data analysis, quality assurance, and predictive modeling. Markov models are an effective tool for time-series data analysis. Previous approaches utilizing Markov models incorporate transition matrices (TMs) at varying dimensionalities and resolutions. Other analysis methods treat TMs as vectors for comparison using search algorithms such as the nearest neighbors comparison algorithm, or use TMs to calculate the probability of discrete subsets of time-series data. We propose an analysis method that treats the elements of a TM as random variables, parameterizing them hierarchically. This approach creates a metric for determining the “normalcy” of a TM generated from a subset of time-series data. The advantages of this novel approach are discussed in terms of computational efficiency, accuracy of anomaly detection, and robustness when analyzing sparse data. Unlike previous approaches, this algorithm is developed with the expectation of sparse TMs. Accounting for this sparseness significantly improves the detection accuracy of the proposed method. Detection rates in a variety of time-series data types range from (97 % TPR, 2.1 % FPR) to (100 % TPR, <0.1 % FPR) with very small sample sizes (20–40 samples) in data with sparse transition probability matrices.