Modeling, Clustering, and Segmenting Video with Mixtures of Dynamic Textures

• Published in: IEEE transactions on pattern analysis and machine intelligence Vol. 30; no. 5; pp. 909 - 926
• Main Authors: Chan, A.B., Vasconcelos, N.
• Format: Journal Article
• Language: English
• Published: Los Alamitos, CA IEEE 01.05.2008; IEEE Computer Society; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Algorithms; Applied sciences; Artificial Intelligence; Cluster Analysis; Clustering; Clustering algorithms; Computer science; control theory; systems; Computer Simulation; Computer vision; Control theory; Dynamic tests; Dynamic texture; Dynamical systems; Dynamics; Exact sciences and technology; expectation-maximization; Fires; Image Enhancement - methods; Image Interpretation, Computer-Assisted - methods; Information Storage and Retrieval - methods; Kalman filter; Likelihood Functions; linear dynamical systems; Linear systems; Machine learning; Machine learning algorithms; Marine vehicles; Mathematical models; mixture models; Models, Statistical; motion segmentation; Pattern Recognition, Automated - methods; Pattern recognition. Digital image processing. Computational geometry; probabilistic models; Representations; Reproducibility of Results; Sensitivity and Specificity; Studies; Surface layer; temporal textures; Texture; time-series clustering; Vehicle dynamics; video clustering; video modeling; Video Recording - methods; Video sequences; video clustering; temporal textures; video modeling; Kalman filter; Dynamic texture; motion segmentation; time-series clustering; mixture models; expectation-maximization; linear dynamical systems; probabilistic models; Cluster analysis; Parameter estimation; Video signal; Mixture theory; Linear time; Computer control; Dynamical system; Modeling; Texture; Image sequence; Fires; Classification; Dynamic model; Pattern analysis; Computer vision; Statistical analysis; Probabilistic approach; Motion estimation; Computer theory; Time series; Pedestrian traffic; Water vapor; Image segmentation; Road traffic; Control theory; EM algorithm; Artificial intelligence; Optical flow
• ISSN: 0162-8828; 2160-9292; 1939-3539
• DOI: 10.1109/TPAMI.2007.70738

A dynamic texture is a spatio-temporal generative model for video, which represents video sequences as observations from a linear dynamical system. This work studies the mixture of dynamic textures, a statistical model for an ensemble of video sequences that is sampled from a finite collection of visual processes, each of which is a dynamic texture. An expectation-maximization (EM) algorithm is derived for learning the parameters of the model, and the model is related to previous works in linear systems, machine learning, time- series clustering, control theory, and computer vision. Through experimentation, it is shown that the mixture of dynamic textures is a suitable representation for both the appearance and dynamics of a variety of visual processes that have traditionally been challenging for computer vision (for example, fire, steam, water, vehicle and pedestrian traffic, and so forth). When compared with state-of-the-art methods in motion segmentation, including both temporal texture methods and traditional representations (for example, optical flow or other localized motion representations), the mixture of dynamic textures achieves superior performance in the problems of clustering and segmenting video of such processes.