A New Memory Based on Sequence to Sequence Model for Video Captioning

• Published in: 2021 International Conference on Security, Pattern Analysis, and Cybernetics（SPAC pp. 470 - 476
• Main Authors: Lin, Jin-Cheng, Zhang, Chun-Yang
• Format: Conference Proceeding
• Language: English
• Published: IEEE 18.06.2021
• Subjects: Attention Module; Correlation; Deep learning; Feature extraction; NTM; Representation learning; Security; Sequence to Sequence; Supervised learning; Training; Video Captioning; Visualization
• DOI: 10.1109/SPAC53836.2021.9539903

In the field of computer vision, Video captioning is a very important and meaningful task, which could automatically generate textual descriptions of contents from videos. It is a challenging problem due to the difficulties of understanding the objects and activities in a video. Benefit from the rapid development of deep learning technology, e.g. sequence to sequence model, video captioning task has achieved very accurate results. However, there are two serious flaws, the first is that the pre-trained deep models are often used as visual feature abstractors as the training is highly time-consuming, so the feature generalization performance generated by these pre-trained Encoder is limited when we directly employ those networks in video captioning tasks. The second is that each frame in the video is processed separately, ignoring the correlation of video data in the time dimension. In this work, we propose video captioning model with attention-memory module to explore the role of capturing temporal correlations which with sequence to sequence model as the background and showing the importance of temporal structure to vision tasks by adding the correlation of videos when extracting features and enhancing the time-memory capability. Our experiments are based on two most famous benchmark datasets in the field of video captioning: MSVD and MSR-VTT. Then employ BLEU and METEOR to evaluate the accuracy of the description generated by different methods. Finally, the experimental results confirm that the proposed model could make significant improvements in description results compared with the baseline models.