Abstract
The last few years have witnessed an exponential growth of data available online in heterogeneous modalities, e.g., images, tags, texts, videos. Current search engines primarily use text tags to retrieve data with similar tags. However, the retrieval performance of such systems is strongly affected by the quality of annotations. Hence, it is desirable to develop efficient multi-media search systems which retrieve data based on the information the data contains. However, developing such multi-media search systems is a challenging task requiring addressing multiple problems. Specifically, (i) learning to extract more discriminative representations, (ii) learning the binary representations which can well preserve the information of the real-value representations (binary representations are necessary for efficient large-scale retrieval), (iii) learning representations of different modalities which can preserve the inter-modal similarity in a common space (i.e., minimizing modality gap), and (iv) improving the computation and memory efficiency of the hashing function. This thesis is dedicated to developing novel methods to address the problems of multi-media search systems. First, the thesis introduces two novel methods to enhancing the discrimination power of CNN features. In the first work, we propose a method which maps a single feature into low-dimensional features, while enhancing the discriminability. In the second work, we present a novel, optimized and computationally-efficient framework for image retrieval task. More specifically, inspired by the fact that outputs of convolutional layers are more useful for the retrieval tasks than the fully-connected features are, we first propose a novel method to compute masks to select a representative subset of local convolutional features. Additionally, we propose to employ recent embedding and aggregating methods which can significantly boost the feature discriminability. Second, to address the second challenge, we revisit the traditional hashing approach, which requires two steps: dimension reduction and binary quantization. However, in the first step, no binary constraint is considered, which might result in suboptimal solutions. Hence, we propose a novel formulations to simultaneously compress the input features (while retained a high amount of variance) and minimize the binary quantization error of projected data in a single projection step. We additionally propose novel algorithms to efficiently solve the challenging non-convex problems with binary and orthogonal constraints. iii Third, we propose two novel methods which tackle the problem of learning binary representations which can preserve both intra and inter-modal similarities for efficient cross-modal retrieval. In the first work, we propose a two step-hashing method to decouple the binary code learning and hashing function learning. In the first step, we propose a novel spectral embedding-based algorithm, which is well-known approach to discover the neighborhood structure of data, to jointly learn single modal representations (to preserve intra-modal similarity) and cross-modal representations (to preserve both intra- and inter-modal similarity). In the second step, to learn hashing functions which map informative data inputs (images and word embeddings) to binary codes obtained from the first step, we leverage the powerful CNN for images and propose a CNN-based deep architecture for text modality. In the second work, we adopt the maximizing mutual information (MI) approach. More specifically, we leverage the recent advances in estimating variational lower-bound of MI to maximizing the MI between the binary representations and input features and between binary representations of different modalities. Maximizing these MI can help to learn binary representations that can preserve both intra- and inter-modal similarity. Additionally, we find out that trying to minimize modality gap excessively by learning similar binary representations for the same instance from different modalities could result in less informative representations. Hence, balancing between reducing modality gap and losing modality-private information is important for the cross-modal retrieval tasks. Finally, to improving the efficiency of hashing functions during the inference stage for both computation and memory aspects, we adopt the network quantization approach. The general idea is to represent each full-precision weight/activation values (32-bits) byK-bits. Quantization weights and activations would result in smaller memory and computation cost as the dot-product (the core operation of convolutional layers) can be computed using bitwise operations (and,xnor, bitcount). To effectively train deep convolutional neural networks with low bit-width weights and activations, we propose two novel techniques, namely Learned Quantized Weights and Activation Channel-Wise Averaged Quantizer, which respectively address the gradient mismatch problem of previous methods in weight updating and the bias issue towards high values in activation quantization.