Multi-Modal Fake News Detection via Bridging the Gap between Modals

• Published in: Entropy (Basel, Switzerland) Vol. 25; no. 4; p. 614
• Main Authors: Liu, Peng, Qian, Wenhua, Xu, Dan, Ren, Bingling, Cao, Jinde
• Format: Journal Article
• Language: English
• Published: Switzerland MDPI AG 04.04.2023; MDPI
• Subjects: Alliances; Analysis; caption-based; Computational linguistics; Disinformation; fake news detection; False information; Image enhancement; Information technology; Language processing; multi-modal; Natural language interfaces; Natural language processing; Neural networks; Semantics; Social networks; transformer; Germany; transformer; caption-based; fake news detection; multi-modal
• ISSN: 1099-4300; 1099-4300
• DOI: 10.3390/e25040614

Multi-modal fake news detection aims to identify fake information through text and corresponding images. The current methods purely combine images and text scenarios by a vanilla attention module but there exists a semantic gap between different scenarios. To address this issue, we introduce an image caption-based method to enhance the model’s ability to capture semantic information from images. Formally, we integrate image description information into the text to bridge the semantic gap between text and images. Moreover, to optimize image utilization and enhance the semantic interaction between images and text, we combine global and object features from the images for the final representation. Finally, we leverage a transformer to fuse the above multi-modal content. We carried out extensive experiments on two publicly available datasets, and the results show that our proposed method significantly improves performance compared to other existing methods.