Abstract
Entropy and its related concepts are widely used in machine learning. In the first part of the thesis, we present an entropy-based framework to estimate the noise transition matrix of a dataset with instance-independent label noise. We proved that our estimator is statistically consistent. To the best of our knowledge, this is the first-ever proof of consistency for any estimator of the noise transition matrix, which means that our estimator is the first-ever estimator of the noise transition matrix that can achieve vanishing estimation error. At the heart of our framework is a discriminator that predicts whether an input dataset D has maximum Shannon entropy, which shall be used on multiple new datasets ˆD synthesized from D via the insertion of additional label noise. Furthermore, our framework has the added benefit of not requiring clean data, or easy-to-identify data, and we do not need to optimize classification accuracy to achieve the vanishing estimate error. A key technical ingredient of our consistency proof is the idea of typical sets, used in what is known in Information Theory as the Asymptotic Equipartition Property (AEP). In our proof, we used a Bayesian analog of joint AEP. We also provide concrete realization of our framework by incorporating local intrinsic dimensionality (LID) into the discriminator. The second part of the thesis focuses on web image datasets with instance-dependent label noise. Such datasets curated online inevitably contain ambiguous in-distribution (ID) instances and out-of-distribution (OOD) instances, which we collectively call nonconforming (NC) instances. Recent methods that tackle the detrimental effect of NC instances have a common underlying idea that these NC instances can be identified via entropy maximization. To make sense of the “entropy” of an instance, we interpret the output prediction vector of an instance probabilistically as the parameter vector of a multinomial random variable, with respect to some trained model. Hence, entropy maximization is based on the idealized assumption that NC instances have predictions that are “almost” uniformly distributed. However, in real-world web image datasets, there are numerous NC instances whose predictions are far from being uniformly distributed. To tackle the limitation of entropy maximization, we propose (a, ß)-generalized KL divergence, Da,ß KL (p?q), which can be used to identify more NC instances. Theoretical properties of Da,ß KL (p?q) are proven, and we also show empirically that a simple use of Da,ß KL (p?q) outperforms all baselines on the NC instance identification task. Building upon Da,ßKL (p?q), we also introduce a new iterative training framework, GENKL, that identifies and relabels NC instances. We achieved new state-of-the-art accuracies:81.34% and 85.73%, on Clothing1M and Food101/Food101N, respectively.