Title: What's next? Vertical Federated Learning

Abstract

Federated learning (FL) enables the training of machine learning models among multiple participants without requiring the transmission or sharing of their data. This is in sharp contrast to traditional machine learning that requires all data to be transmitted to a central place. Privacy requirements and regulation have been two important drivers of FL, making it a popular approach for creating machine learning models with data that would be otherwise unavailable for training. Federated learning can be classified into vertical and horizontal, depending on what data is available at each party. Horizontal FL has by far dominated the research space; in horizontal FL, all participants have access to the same type of data and can each create a model in isolation. In contrast, vertical FL has received far less attention despite its utility in real life use cases. In vertical FL each party collects and contributes diverse knowledge and not all participants need to have a label available. This makes it suitable for scenarios where each party cannot train a model in isolation or where ground truth data (labels) are not widely available. In this talk, I will start by defining what is vertical FL as there are multiple definitions in the literature. I will then present some interesting use cases, contrasting the requirements and pragmatics of vertical FL. I will also cover some of my recent work in vertical FL showcasing how different privacy requirements can shape drastically different solutions. Finally, I will conclude showing some open challenges in this area.

Title: Advances in Private Cross-Device Federated Learning

Abstract

Privacy for users is a central goal of cross-device federated learning. This talk will begin with a broad view of privacy, highlighting key principles and threat models. We will then deep-dive into some recent advances in providing stronger anonymization properties for cross-device federated learning, including the DP-FTRL algorithm. This algorithm has been deployed successfully at scale, enabling the training and launch of production neural language models trained with user-level differential privacy guarantees. We will also discuss the `federated select` primitive, which provides a useful abstraction for incorporating private information retrieval into federated computations, enabling a variety of new applications.

Title: Bridging the Gap: Federated Learning for Large Language Models and Information Retrieval

Abstract

We will explore the intersection of Federated Machine Learning (FML), Large Language Models (LLMs), Information Retrieval (IR), and Search, with a focus on edge computing. The talk will cover decentralized LLM training, efficient model updates, synchronization, and personalization. We will delve into privacy-preserving search systems, relevance feedback, and query expansion in federated settings. We will start by reviewing recent work in model quantization for FML to efficiently and effectively train a neural network model. We will then highlight the risk of adversarial attacks in federated learning systems, such as Byzantine attacks and discuss potential defense mechanisms. The presentation will also introduce machine unlearning as a promising avenue for future research in federated learning

Title: Evaluating Large-Scale Learning Systems

Abstract

To deploy machine learning models in practice it is critical to have a way to reliably evaluate their effectiveness. Unfortunately, the scale and complexity of modern machine learning systems make it difficult to provide faithful evaluations and gauge performance across potential deployment scenarios. In this talk, I discuss our work addressing challenges in large-scale ML evaluation. First, I explore the problem of evaluating models trained in federated networks of devices, where issues of device subsampling, heterogeneity, and privacy can introduce noise in the evaluation process and make it challenging to provide reliable evaluations. Second, I present ReLM, a system for validating and querying large language models (LLMs). Although LLMs have been touted for their ability to generate natural-sounding text, there is a growing need to evaluate the behavior of LLMs in light of issues such as data memorization, bias, and inappropriate language. ReLM poses LLM validation queries as regular expressions to enable faster and more effective LLM evaluation.