Deep correspondence restricted Boltzmann machine for cross-modal retrieval

作者：Feng, Fangxiang; Li, Ruifan^*; Wang, Xiaojie

来源：Neurocomputing, 2015, 154: 50-60.

DOI：10.1016/j.neucom.2014.12.020

• Cross-modal
• RBM
• Retrieval
• Deep Learning
• Multi-modal

The task of cross-modal retrieval, i.e., using a text query to search for images or vice versa, has received considerable attention with the rapid growth of multi-modal web data. Modeling the correlations between different modalities is the key to tackle this problem. In this paper, we propose a correspondence restricted Boltzmann machine (Corr-RBM) to map the original features of bimodal data, such as image and text in our setting, into a low-dimensional common space, in which the heterogeneous data are comparable. In our Corr-RBM, two RBMs built for image and text, respectively are connected at their individual hidden representation layers by a correlation loss function. A single objective function is constructed to trade off the correlation loss and likelihoods of both modalities. Through the optimization of this objective function, our Corr-RBM is able to capture the correlations between two modalities and learn the representation of each modality simultaneously. Furthermore, we construct two deep neural structures using Corr-RBM as the main building block for the task of cross-modal retrieval. A number of comparison experiments are performed on three public real-world data sets. All of our models show significantly better results than state-of-the-art models in both searching images via text query and vice versa.

• 出版日期2015-4-22
• 单位北京邮电大学