Federated Reinforcement Learning: Statistical and Communication Trade-offs

Reinforcement learning (RL), concerning decision making in uncertain environments, lies at the heart of modern artificial intelligence. Due to the high dimensionality, training of RL agents typically requires a significant amount of computation and data to achieve desirable performance. However, data collection can be extremely time-consuming with limited access in real-world applications, especially when performed by a single agent. On the other hand, it is plausible to leverage multiple agents to collect data simultaneously, under the premise that they can learn a global policy collaboratively without the need of sharing local data in a federated manner. This talk addresses the fundamental statistical and communication trade-offs in the algorithmic designs of federated RL algorithms, covering both blessings and curses in the presence of data and task heterogeneities across the agents. 

Bio: Dr. Yuejie Chi is the Sense of Wonder Group Endowed Professor of Electrical and Computer Engineering in AI Systems at Carnegie Mellon University, with courtesy appointments in the Machine Learning department and CyLab. She received her Ph.D. and M.A. from Princeton University, and B. Eng. (Hon.) from Tsinghua University, all in Electrical Engineering. Her research interests lie in the theoretical and algorithmic foundations of data science, signal processing, machine learning and inverse problems, with applications in sensing, imaging, decision making, and AI systems. Among others, Dr. Chi received the Presidential Early Career Award for Scientists and Engineers (PECASE), SIAM Activity Group on Imaging Science Best Paper Prize, IEEE Signal Processing Society Young Author Best Paper Award, and the inaugural IEEE Signal Processing Society Early Career Technical Achievement Award for contributions to high-dimensional structured signal processing. She is an IEEE Fellow (Class of 2023) for contributions to statistical signal processing with low-dimensional structures.

You Trained with Offline RL---But How to Tune It?

Given two candidate functions, can we identify which is the true value function of a large Markov decision process (MDP), given a "benign" dataset? Trivial as it might seem, a version of the question was open for 20+ years in reinforcement learning (RL). The lack of satisfying solutions to such a simple question is also why we are not seeing wide deployment of offline RL in real-world applications, as this is a basic model-selection/hyperparameter-tuning question that is impossible to avoid in practice. In this talk, I will discuss how the core technical difficulties are intimately related to those behind the training instability of modern deep RL. By bridging the perspectives of training stability and model selection, I will show how our framework, Batch Value-Function Tournament (BVFT), overcomes the long-perceived theoretical barrier and demonstrates promising empirical performance.

Bio: Nan Jiang is an Associate Professor of Computer Science at University of Illinois at Urbana-Champaign. Prior to joining UIUC, he was a postdoc researcher at Microsoft Research NYC. He received his PhD in Computer Science and Engineering at University of Michigan. His research focuses on the theory of reinforcement learning, with specific interests in the sample complexity of exploration under function approximation, offline RL and evaluation, and learning in partially observable systems. He coauthors a monograph on RL theory and holds editorial positions in the research community, including serving as an action editor for JMLR, an editor for FnT in ML, and Senior Area Chairs for ICML and ICLR. His contributions are recognized by  Best Paper Award in AAMAS 2015, Outstanding Paper Runner-up in ICML 2022, Adobe Data Science Award in 2021, NSF CAREER Award in 2022, Google Research Scholarship in 2024, and Sloan Research Fellowship in 2024.

Optimizing Data for Decision-Making

We study the fundamental question of how informative a dataset is for solving a given decision-making task. In our setting, the dataset provides partial information about unknown parameters that influence task outcomes. Focusing on linear programs, we characterize when a dataset is sufficient to recover an optimal decision, given an uncertainty set on the cost vector. Our main contribution is a sharp geometric char- acterization that identifies the directions of the cost vector that matter for optimality, relative to the task constraints and uncertainty set. We further develop a practical algorithm that, for a given task, constructs a minimal or least-costly sufficient dataset. Our results reveal that small, well-chosen datasets can often fully determine optimal decisions—offering a principled foundation for task-aware data selection. 

Bio: Professor Ozdaglar’s research focuses on technical and societal aspects of large-scale data-driven systems. Her expertise includes optimization, machine learning, economics, and networks. In recent years, she has investigated issues of data ownership and markets, spread of misinformation on social media, economic and financial contagion, and social learning. In addition, she has an active research program on large-scale optimization, especially in the context of machine learning. Her recent work develops robust, efficient and decentralized machine learning models and algorithms. In optimization, her work has contributed to optimization duality, first-order scalable methods and new distributed algorithms for network resource allocation. Her work has also contributed to the study of adaptive game-theoretic dynamics and introduced alternative approaches to network games.

Foundation Models for Autonomous Vehicles

Foundation models, trained on vast and diverse data encompassing the human experience, are at the heart of the ongoing AI revolution influencing the way we create, problem solve, and work. These models, and the lessons learned from their construction, can also be applied to the way we develop a similarly transformative technology, autonomous vehicles (AVs). In this talk, I will highlight recent research efforts towards rethinking elements of an AV program both in the vehicle and in the data center, with an emphasis on (1) composing ingredients for universal and controllable end-to-end simulation, (2) architecting autonomy stacks that leverage foundation models to generalize to long-tail events, and (3) ensuring safety with foundation models in the loop.

Bio: Dr. Marco Pavone is an Associate Professor of Aeronautics and Astronautics at Stanford University, where he is the Director of the Autonomous Systems Laboratory and Co-Director of the Center for Automotive Research at Stanford. Before joining Stanford, he was a Research Technologist within the Robotics Section at the NASA Jet Propulsion Laboratory. He received a Ph.D. degree in Aeronautics and Astronautics from the Massachusetts Institute of Technology in 2010. His main research interests are in the development of methodologies for the analysis, design, and control of autonomous systems, with an emphasis on self-driving cars, autonomous aerospace vehicles, and future mobility systems. He is a recipient of a number of awards, including a Presidential Early Career Award for Scientists and Engineers from President Barack Obama, an Office of Naval Research Young Investigator Award, a National Science Foundation Early Career (CAREER) Award, a NASA Early Career Faculty Award, and an Early-Career Spotlight Award from the Robotics Science and Systems Foundation. He was identified by the American Society for Engineering Education (ASEE) as one of America's 20 most highly promising investigators under the age of 40. His work has been recognized with best paper nominations or awards at the European Control Conference, at the IEEE International Conference on Intelligent Transportation Systems, at the Field and Service Robotics Conference, at the Robotics: Science and Systems Conference, at the ROBOCOMM Conference, and at NASA symposia. He is currently serving as an Associate Editor for the IEEE Control Systems Magazine. He is serving or has served on the advisory board of a number of autonomous driving start-ups (both small and multi-billion dollar ones), he routinely consults for major companies and financial institutions on the topic of autonomous systems, and is a venture partner for investments in AI-enabled robots.

Some variants of gradient dominance conditions motivated by LQR direct policy optimization

Solutions of optimization problems, including policy optimization in reinforcement learning, typically rely upon some variant of gradient descent. There has been much recent work in the machine learning, control, and optimization communities applying the Polyak-Łojasiewicz Inequality (PŁI) to such problems in order to establish an exponential rate of convergence  (a.k.a. “linear convergence” in the local-iteration language of numerical analysis) of loss functions to their minima under the gradient flow. Often, as is the case of policy iteration for the continuous-time LQR problem, this rate vanishes for large initial conditions, resulting in a mixed globally linear / locally exponential behavior. This is in sharp contrast with the discrete-time LQR problem, where there is global exponential convergence. That gap between CT and DT behaviors motivates the search for various generalized PŁI-like conditions, and this talk will address that topic. Moreover, these generalizations are key to understanding the transient and asymptotic effects of errors in the estimation of the gradient, errors which might arise from adversarial attacks, wrong evaluation by an oracle, early stopping of a simulation, inaccurate and very approximate digital twins, stochastic computations (algorithm "reproducibility"), or learning by sampling from limited data. We will describe an “input to state stability” (ISS) analysis of this issue. We will also discuss convergence and PŁI-like properties of “linear feedforward neural networks” in feedback control.  (Joint work with A.C.B. de Oliveira, L. Cui, Z.P. Jiang, and M. Siami).

BIO: Eduardo D. Sontag received his Licenciado in Mathematics at the University of Buenos Aires (1972) and a Ph.D. in Mathematics (1977) under Rudolf E. Kalman at the University of Florida. From 1977 to 2017, he was at Rutgers University, where he was a Distinguished Professor of Mathematics and a Member of the Graduate Faculty of the Departments of Computer Science and of Electrical and Computer Engineering and the Cancer Institute of NJ. He directed the undergraduate Biomathematics Interdisciplinary Major and the Center for Quantitative Biology, and was Graduate Director at the Institute for Quantitative Biomedicine. In January 2018, Dr. Sontag became a University Distinguished Professor in the Departments of Electrical and Computer Engineering and of BioEngineering at Northeastern University, where he is also affiliated with the Mathematics and the Chemical Engineering departments. Since 2006, he has been a Research Affiliate at the Laboratory for Information and Decision Systems, MIT, and since 2018 he has been a Faculty Member in the Program in Therapeutic Science at Harvard Medical School. His major current research interests lie in several areas of control and dynamical systems theory, systems molecular biology, cancer and immunology, machine learning, and computational biology. Sontag was awarded the Reid Prize in Mathematics in 2001, the 2002 Hendrik W. Bode Lecture Prize and the 2011 Control Systems Field Award from the IEEE, the 2022 Richard E. Bellman Control Heritage Award, the 2023 IFAC Triennial Award on Nonlinear Control, the 2002 Board of Trustees Award for Excellence in Research from Rutgers, and the 2005 Teacher/Scholar Award from Rutgers. Sontag is a Fellow of IEEE, AMS, SIAM, and IFAC. In 2024 he was inducted into the American Academy of Arts and Sciences, and in 2025 he was elected into the National Academy of Sciences.

If Gen AI is the answer, what is the question? Some thoughts on the theory of generation

Generative AI refers to AI techniques that learn to generate outputs such as text, images, audio, and video--often in response to user prompts. There is a fundamental tension in generative AI between generating outputs that resemble the training data versus generating truly novel content. While researchers have proposed many innovative generative AI methods, comparatively less attention has been given to articulating what the problem is. Most existing work uses probabilistic foundations even though the specification of the generation task arguably does not refer to probability at all.

In this talk, I will focus on a key question: what are the cleanest abstractions that capture the essence of generation? In particular, I will focus on a probability-free model of generation recently proposed by Kleinberg and Mullainathan. I will talk about both the history of the model as well as follow-up work that extends the basic model in various interesting ways. I will conclude by pointing out several fruitful avenues for further work.

Bio Ambuj Tewari is a professor in the Department of Statistics and the Department of EECS (by courtesy) at the University of Michigan, Ann Arbor. He is also affiliated with the Michigan Institute for Data & AI in Society (MIDAS). His research interests lie in machine learning including statistical learning theory, online learning, reinforcement learning and control theory, network analysis, and optimization for machine learning. He collaborates with scientists to seek novel applications of machine learning in behavioral sciences, psychiatry, and chemical sciences. His research has been recognized with paper awards at COLT 2005, COLT 2011, AISTATS 2015, and ALT 2025. He received an NSF CAREER award in 2015, a Sloan Research Fellowship in 2017, and an Early Career Award in Statistics and Data Sciences by the International Indian Statistical Association in 2023. He was named an IMS Fellow in 2022.