Hypergraph-Based Multi-View Action Recognition Using Event Cameras

• Published in: IEEE transactions on pattern analysis and machine intelligence Vol. 46; no. 10; pp. 6610 - 6622
• Main Authors: Gao, Yue, Lu, Jiaxuan, Li, Siqi, Li, Yipeng, Du, Shaoyi
• Format: Journal Article
• Language: English
• Published: United States IEEE 01.10.2024
• Subjects: Cameras; dynamic vision sensor; event camera; Feature extraction; hypergraph neural network; Multi-view action recognition; Neural networks; Robot vision systems; Semantics; Task analysis; Vision sensors
• ISSN: 0162-8828; 1939-3539; 2160-9292; 1939-3539
• DOI: 10.1109/TPAMI.2024.3382117

Action recognition from video data forms a cornerstone with wide-ranging applications. Single-view action recognition faces limitations due to its reliance on a single viewpoint. In contrast, multi-view approaches capture complementary information from various viewpoints for improved accuracy. Recently, event cameras have emerged as innovative bio-inspired sensors, leading to advancements in event-based action recognition. However, existing works predominantly focus on single-view scenarios, leaving a gap in multi-view event data exploitation, particularly in challenges like information deficit and semantic misalignment. To bridge this gap, we introduce HyperMV , a multi-view event-based action recognition framework. HyperMV converts discrete event data into frame-like representations and extracts view-related features using a shared convolutional network. By treating segments as vertices and constructing hyperedges using rule-based and KNN-based strategies, a multi-view hypergraph neural network that captures relationships across viewpoint and temporal features is established. The vertex attention hypergraph propagation is also introduced for enhanced feature fusion. To prompt research in this area, we present the largest multi-view event-based action dataset <inline-formula><tex-math notation="LaTeX">\mathbf{THU}^{\mathbf{MV-EACT}}\mathbf{-50}</tex-math> <mml:math><mml:mrow><mml:msup><mml:mi mathvariant="bold">THU</mml:mi><mml:mrow><mml:mi mathvariant="bold">MV</mml:mi><mml:mo>-</mml:mo><mml:mi mathvariant="bold">EACT</mml:mi></mml:mrow></mml:msup><mml:mo>-</mml:mo><mml:mn mathvariant="bold">50</mml:mn></mml:mrow></mml:math><inline-graphic xlink:href="lu-ieq1-3382117.gif"/> </inline-formula>, comprising 50 actions from 6 viewpoints, which surpasses existing datasets by over tenfold. Experimental results show that HyperMV significantly outperforms baselines in both cross-subject and cross-view scenarios, and also exceeds the state-of-the-arts in frame-based multi-view action recognition.