Federated Learning One World Seminar

Abstract: Distributed learning has emerged as a leading paradigm for training large machine learning models. However, in real-world scenarios, participants may be unreliable or malicious, posing a significant challenge to the integrity and accuracy of the trained models. Byzantine fault tolerance mechanisms have been proposed to address these issues, but they often assume full participation from all clients, which is not always practical due to the unavailability of some clients or communication constraints. In our work, we propose the first distributed method with client sampling and provable tolerance to Byzantine workers. The key idea behind the developed method is the use of gradient clipping to control stochastic gradient differences in recursive variance reduction. This allows us to bound the potential harm caused by Byzantine workers, even during iterations when all sampled clients are Byzantine. Furthermore, we incorporate communication compression into the method to enhance communication efficiency. Under quite general assumptions, we prove convergence rates for the proposed method that match the existing state-of-the-art (SOTA) theoretical results.

Abstract: Federated learning is a powerful distributed optimization framework in which multiple clients collaboratively train a global model without sharing their raw data. In this work, we tackle the personalized version of the federated learning problem. In particular, we ask: throughout the training process, can each client in a federated system identify a subset of similar clients and collaboratively train with just those clients? In the affirmative, we formalize this problem as a stochastic optimization problem, achieving optimal convergence rates for a large class of loss functions. We propose simple iterative algorithms which identify clusters of similar clients and train a personalized model-per-cluster, using local client gradients and flexible constraints on the clusters. The convergence rates of our algorithms asymptotically match those obtained if we knew the true underlying clustering of the clients and are provably robust in the Byzantine setting where some fraction of the clients are malicious.

Abstract: Data sharing and collaborative (machine) learning have use-cases where individual "agents" (e.g., researchers, organizations) have limited capabilities or resources to conduct large-scale data collection and thus turn to each other for collaboration. A motivating example is in medicine (NEJM, Nature Journal) where the data are incredibly valuable and costly, and are under stringent privacy regulations. Hence, we formally study the problem of data sharing and/or collaborative learning to analyze the desiderata (in light of these practical considerations) and propose principled solutions to achieve them. We identify several important desiderata, especially the fairness of the collaboration, which is motivated by the fact that each agent incurs a non-trivial (and sometimes significant) cost from procuring, acquiring or otherwise collecting their data, so it is imperative that their effort is fairly recognized and rewarded, in the form of some specific incentives. In this talk, I will talk about some precise formalizations of fairness (e.g., the Shapley value), how it is applied to incentivize data sharing/collaborative learning in some specific learning contexts (e.g., federated learning), and discuss some future directions.

Abstract: A main challenge in many practical scenarios of federated learning is that the clients are only intermittently available to participate in learning. In this talk, I will present our recent results on understanding and overcoming this challenge. I will first explain the importance of aggregation weight adaptation and introduce a new algorithm that improves federated averaging (FedAvg) by adaptively weighting the client updates. The adaptation is based on online estimates of the optimal weights, where the statistics of client participation are heterogeneous and unknown a priori. Then, for the case with very infrequently participating clients, I will present an "amplification" mechanism that is applied to the model updates. The talk will cover both theoretical and empirical findings on this topic and also discuss further insights.