Learning unified binary codes for cross-modal retrieval via latent semantic hashing

• Published in: Neurocomputing (Amsterdam) Vol. 213; pp. 191 - 203
• Main Authors: Xu, Xing, He, Li, Shimada, Atsushi, Taniguchi, Rin-ichiro, Lu, Huimin
• Format: Journal Article
• Language: English
• Published: Elsevier B.V 12.11.2016
• Subjects: Binary representation; Cross-modal retrieval; Hashing; Matrix factorization; Sparse coding; Hashing; Cross-modal retrieval; Matrix factorization; Sparse coding; Binary representation
• ISSN: 0925-2312; 1872-8286
• DOI: 10.1016/j.neucom.2015.11.133

Nowadays the amount of multimedia data such as images and text is growing exponentially on social websites, arousing the demand of effective and efficient cross-modal retrieval. The cross-modal hashing based methods have attracted considerable attention recently as they can learn efficient binary codes for heterogeneous data, which enables large-scale similarity search. Generally, to effectively construct the cross-correlation between different modalities, these methods try to find a joint abstraction space where the heterogeneous data can be projected. Then a quantization rule is applied to convert the abstraction representation to binary codes. However, these methods may not effectively bridge the semantic gap through the latent abstraction space because they fail to capture latent information between heterogeneous data. In addition, most of these methods apply the simplest quantization scheme (i.e. sign function) which may cause information loss of the abstraction representation and result in inferior binary codes. To address these challenges, in this paper, we present a novel cross-modal hashing based method that generates unified binary codes combining different modalities. Specifically, we first extract semantic features from the modalities of images and text to capture latent information. Then these semantic features are projected to a joint abstraction space. Finally, the abstraction space is rotated to produce better unified binary codes with much less quantization loss, while preserving the locality structure of projected data. We integrate the binary code learning procedures above to develop an iterative algorithm for optimal solutions. Moreover, we further exploit the useful class label information to reduce the semantic gap between different modalities to benefit the binary code learning. Extensive experiments on four multimedia datasets show that the proposed binary coding schemes outperform several other state-of-the-art methods under cross-modal scenarios.