Featured correspondence topic model for semantic search on social image collections

• Published in: Expert systems with applications Vol. 77; pp. 20 - 33
• Main Authors: Tu, Nguyen Anh, Khan, Kifayat Ullah, Lee, Young-Koo
• Format: Journal Article
• Language: English
• Published: New York Elsevier Ltd 01.07.2017; Elsevier BV
• Subjects: Digital imaging; Digital media; Expert systems; Image annotation; Image retrieval; Knowledge; Mathematical models; Object recognition; Probabilistic graphical model; Retrieval; Searching; Semantic analysis; Similarity; Social image tagging; Tags; Topic modeling; Topic modeling; Probabilistic graphical model; Social image tagging; Image retrieval; Image annotation
• ISSN: 0957-4174; 1873-6793
• DOI: 10.1016/j.eswa.2017.01.055

•A new framework to retrieve semantically relevant images from the social database.•Probabilistic topic model to predict the missing tags and remove the noisy ones.•Two algorithms for the estimation of model parameters and tag correspondence.•The scoring scheme relies on the fusion of visual and textual information.•The outperformance of image annotation and retrieval to state-of-the-art methods. Nowadays, due to the rapid growth of digital technologies, huge volumes of image data are created and shared on social media sites. User-provided tags attached to each social image are widely recognized as a bridge to fill the semantic gap between low-level image features and high-level concepts. Hence, a combination of images along with their corresponding tags is useful for intelligent retrieval systems, those are designed to gain high-level understanding from images and facilitate semantic search. However, user-provided tags in practice are usually incomplete and noisy, which may degrade the retrieval performance. To tackle this problem, we present a novel retrieval framework that automatically associates the visual content with textual tags and enables effective image search. To this end, we first propose a probabilistic topic model learned on social images to discover latent topics from the co-occurrence of tags and image features. Moreover, our topic model is built by exploiting the expert knowledge about the correlation between tags with visual contents and the relationship among image features that is formulated in terms of spatial location and color distribution. The discovered topics then help to predict missing tags of an unseen image as well as the ones partially labeled in the database. These predicted tags can greatly facilitate the reliable measure of semantic similarity between the query and database images. Therefore, we further present a scoring scheme to estimate the similarity by fusing textual tags and visual representation. Extensive experiments conducted on three benchmark datasets show that our topic model provides the accurate annotation against the noise and incompleteness of tags. Using our generalized scoring scheme, which is particularly advantageous to many types of queries, the proposed approach also outperforms state-of-the-art approaches in terms of retrieval accuracy.