Abstract
Federated learning is proposed by Google to provide decentralized intelligent services while addressing data isolation and user privacy issues. The key idea is that clients can collaboratively train machine learning models without sharing data. As federated learning achieves collaborative learning in a privacy-preserving manner, it attracts increasingly more attention from both industry and academia. However, in real-world federated learning implementations over heterogeneous networks, learning performance would be significantly constrained by data heterogeneity and limited communications. In this thesis, we shall improve the generalization and communication efficiency performance of federated learning systems, so as to combat the above challenges. Specifically, we focus on client selection, model updates between the clients and server, as well as learning with real-world data, where data quality is an important measure of data heterogeneity. The thesis begins with an analysis of client selection in federated learning systems. We developed a distance-based attention mechanism to capture the heterogeneity of local training progress via global-local model divergence, and we also proposed a dynamic fraction method for client selection by considering the trade-off between the training cost (i.e., communication cost and computation cost) and the training performance (i.e., training stability and convergence). To the best of our knowledge, we were the first to investigate dynamic fractions for federated learning. We showed that outperformance with respect to model accuracy, stability, and communication efficiency could be achieved by our proposed AdaFL method. Next, we investigated partial model transmission for deep learning (DL)-based federated learning. We found that, with a proper design of partial models, communication efficiency versus model performance is not necessarily a trade-off in federated learning. By dropping the assumption that the transmitted local models must be used for inference, we proposed a subblock omission method to generate partial models for uplink (client-to-server) transmission. This partial model design is motivated by aggregated transformations. An Amalgamation process is proposed to merge diverse partial local models into a complete global model at the server. Moreover, the model omission process throughout training could be dynamically adjusted by a model-divergence-based method. We showed that our proposed Fed-DynAmal could achieve iii better communication efficiency while concurrently achieving better accuracy performance. In the final part of this thesis, we focus on real-world federated data settings. We proposed a general framework for the easy generation of federated synthetic label noise and diverse (e.g. non-IID) client data partitions, so as to simulate real-world data heterogeneity. To tackle the discrepancies in both local data statistics and label qualities, we developed a multi-stage training framework, FedCorr, which consists of pre-processing, finetuning, and usual training stages, where LID scores are used to identify noisy clients, and per-sample losses are adopted for label correction. We demonstrated the outperformance of our proposed FedCorr via extensive experiments with respect to test accuracy and communication efficiency. Keywords: Federated learning, communication efficiency, data heterogeneity, real-world data, heterogeneous network, deep learning, adaptive training.