Multi-Dimensional Attention With Similarity Constraint for Weakly-Supervised Temporal Action Localization

• Published in: IEEE transactions on multimedia Vol. 25; pp. 4349 - 4360
• Main Authors: Chen, Zhengyan, Liu, Hong, Zhang, Linlin, Liao, Xin
• Format: Journal Article
• Language: English
• Published: Piscataway IEEE 2023; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Annotations; Feature extraction; Localization; Location awareness; Multi-dimensional attention; Optical flow; Optical flow (image analysis); Proposals; Segments; Similarity; Task analysis; temporal action localization; video analysis; Videos; weakly supervised learning
• ISSN: 1520-9210; 1941-0077
• DOI: 10.1109/TMM.2022.3174344

Weakly-supervised temporal action localization (WTAL) is a challenging task in understanding untrimmed videos, in which no frame-wise annotation is provided during training, only the video-level category label is available. Current methods mainly adopt temporal attention branches to conduct foreground-background separation with RGB and optical flow features simply concatenated, regardless of the discriminative spacial features and the complementarity between different modalities. In this work, we propose a Multi-Dimensional Attention (MDA) method to explore attention mechanism across three dimensions in weakly supervised action localization, i . e ., 1) temporal attention that focuses on segments containing action instances, 2) channel attention that discovers the most relevant cues for action description, and 3) modal attention that fuses RGB and flow information adaptively based on feature magnitudes during background modeling. In addition, we introduce a similarity constraint loss to refine the action segment representation in feature space, which helps the network to detect less discriminative frames of an action to capture the full action boundaries. The proposed MDA with similarity constraints can be easily applied to existing action detection frameworks with few parameters. Extensive experiments on THUMOS'14 and ActivityNet v1.2 datasets show that the proposed method outperforms the current state-of-the-art WTAL approaches, and achieves comparable results with some advanced fully-supervised methods.