End-to-end Video-level Representation Learning for Action Recognition

• Published in: 2018 24th International Conference on Pattern Recognition (ICPR) pp. 645 - 650
• Main Authors: Zhu, Jiagang, Zhu, Zheng, Zou, Wei
• Format: Conference Proceeding
• Language: English
• Published: IEEE 01.08.2018
• Subjects: Adaptation models; Aggregates; Kinetic theory; Optical imaging; Pattern recognition; Streaming media; Training
• DOI: 10.1109/ICPR.2018.8545710

From the frame/clip-level feature learning to the video-level representation building, deep learning methods in action recognition have developed rapidly in recent years. However, current methods suffer from the confusion caused by partial observation training, or without end-to-end learning, or restricted to single temporal scale modeling and so on. In this paper, we build upon two-stream ConvNets and propose Deep networks with Temporal Pyramid Pooling (DTPP), an end-to-end video-level representation learning approach, to address these problems. Specifically, at first, RGB images and optical flow stacks are sparsely sampled across the whole video. Then a temporal pyramid pooling layer is used to aggregate the frame-level features which consist of spatial and temporal cues. Lastly, the trained model has compact video-level representation with multiple temporal scales, which is both global and sequence-aware. Experimental results show that DTPP achieves the state-of-the-art performance on two challenging video action datasets: UCF101 and HMDB51, either by ImageNet pre-training or Kinetics pre-training.