Abstract
Deep Neural Networks (DNNs) have achieved remarkable success on classification problems attributed to the collection of large-scale datasets. However, it is expensive and time-consuming to obtain annotations with high quality, especially for those real-world datasets. There are two promising ways to mitigate the performance degradation caused by the lack of annotations with high quality: improving the robustness of DNNs toward label noise and exploiting unlabeled data via semi-supervised learning. The goal of this dissertation is to investigate how to tackle the lack of annotation in these two ways.
The first work aims to achieve universal robustness at all noise levels, without making any assumptions on the noise model. We propose a distillation-based framework that incorporates a new subcategory of Positive-Unlabeled learning. In particular, we shall assume that a small subset of any given noisy dataset is known to have correct labels, which we treat as “positive”, while the remaining noisy subset is treated as “unlabeled”. Our framework consists of the following two components: (1) We shall generate, via iterative updates, an augmented clean subset with additional reliable “positive” samples filtered from “unlabeled” samples; (2) We shall train a teacher model on this larger augmented clean set, then we train a student model on the whole dataset, with the guidance of the teacher model. Experiments were conducted on the CIFAR-10 dataset with synthetic label noise at multiple noise levels for both symmetric and asymmetric noise. The results show that our framework generally outperforms at medium to high noise levels. We also evaluated our framework on Clothing1M, a real-world noisy dataset, and we achieved 2.94% improvement in accuracy over existing state-of-the-art methods.
We then investigate the problem of heterogeneous label noise in real-world federated learning (FL) implementations. Federated learning is a privacy-preserving distributed learning paradigm that enables clients to jointly train a global model. In real-world federated learning implementations, client data could have label noise, and different clients could have vastly different label noise levels. Although there exist methods in centralized learning for tackling label noise, such methods do not perform well on heterogeneous label noise in federated learning settings, due to the typically smaller sizes of client datasets and data privacy requirements in federated learning. In this work, we propose FedCorr, a general multi-stage framework to tackle heterogeneous label noise in federated learning, without making any assumptions on the noise models of local clients, while still maintaining client data privacy. Experiments conducted on CIFAR-10/100 with federated synthetic label noise, and on a real-world noisy dataset, Clothing1M, demonstrate that FedCorr is robust to label noise and substantially outperforms the state-of-the-art methods at multiple noise levels.
In the third work, we further extend our study to semi-supervised federated learning with varying distributions of labeled data among clients. In real-world applications, the clients may not be able to label all the data, due to the expensive annotation cost and the diverse expertise and data bias. Hence we consider a more realistic case where each client only collects data from certain (not all) classes. Furthermore, each client only annotates a few samples belonging to a subset of the classes, while the samples in the remaining classes are totally unlabeled. To tackle this problem, we propose a local intrinsic dimensionality (LID)-based metric to adjust the weights of clients for model aggregation and client selection, as well as two data augmentation methods based on frequency features extracted via 2D Discrete Fourier Transform to mitigate the deficiency of labeled data. Experiments conducted on CIFAR-10 demonstrate that our framework is able to improve the performance of federated learning with the problem of deficient labeled data.