Sergey Levine (UC Berkeley)
Title: How do we train foundation models to understand the physical world?
Abstract: Human understanding of the physical world is deeply grounded in our own experience: we are so attuned to physical interactions that we can not only effortlessly perform an enormous range of physical behaviors, but we even use physical metaphors to reason about abstract situations ("this project has a lot of momentum" or, famously, "deep learning is hitting a wall"). On the other hand, the most widely used foundation models are trained on abstract and symbolic data: text, semantic understanding of images, code. What would it take to train foundation models that understand the physical world the way we do, starting with a grounded understanding of physical interactions? In this talk, I'll discuss how robot foundation models can bring us closer to this kind of AI framework, provide an overview of how modern robot foundation models work, and present some results suggesting that this can enable a combined understanding of both physical interaction and semantics together.
Bio: Sergey Levine received a BS and MS in Computer Science from Stanford University in 2009, and a Ph.D. in Computer Science from Stanford University in 2014. He joined the faculty of the Department of Electrical Engineering and Computer Sciences at UC Berkeley in fall 2016. His work focuses on machine learning for decision making and control, with an emphasis on deep learning and reinforcement learning algorithms. Applications of his work include autonomous robots and vehicles, as well as applications in other decision-making domains. His research includes developing algorithms for end-to-end training of deep neural network policies that combine perception and control, scalable algorithms for inverse reinforcement learning, deep reinforcement learning algorithms, and more.
Lu Lu (Yale University)
Title: Physics-informed foundation neural operators and diffusion models over function spaces
Abstract: Deep neural operators have emerged as a powerful paradigm in scientific machine learning (SciML) for learning nonlinear mappings between function spaces arising in complex dynamical systems. In this talk, I will introduce the deep operator network (DeepONet) and its extensions, such as geometry-dependent/manifold operator learning, and demonstrate their effectiveness on diverse multiphysics and multiscale 3D problems, such as geological carbon sequestration, fire simulation, full waveform inversion, and topology optimization. I will then present the first operator learning method requiring only a single PDE solution, i.e., one-shot learning, enabled by a new concept of local solution operator motivated by the locality of physics. Moreover, I will discuss our recent work on diffusion models, including FunDiff as a novel framework of diffusion models over function spaces for physics-informed generative modeling, and RED-DiffEq which leverages denoising diffusion models as regularization for inverse PDE problems. Finally, I will present a physics-informed multimodal foundation model for PDEs and neural-operator element method (an efficient and scalable finite element method enabled by reusable neural operators).
Bio: Lu Lu is an Assistant Professor in the Departments of Statistics and Data Science and of Chemical and Environmental Engineering at Yale University. Prior to joining Yale, he was an Assistant Professor in the Department of Chemical and Biomolecular Engineering at University of Pennsylvania from 2021 to 2023, and an Applied Mathematics Instructor in the Department of Mathematics at Massachusetts Institute of Technology from 2020 to 2021. He obtained his Ph.D. degree in Applied Mathematics at Brown University in 2020, master's degrees in Engineering, Applied Mathematics, and Computer Science at Brown University, and bachelor's degrees in Mechanical Engineering, Economics, and Computer Science at Tsinghua University in 2013. His current research interest lies in scientific machine learning and artificial intelligence for science, including theory, algorithms, software, and its applications to engineering, physical, and biological problems. His broad research interests focus on multiscale modeling and high performance computing for physical and biological systems. He has received the MIT Technology Review Innovators under 35 Asia Pacific, U.S. Department of Energy Early Career Award, Mathematics Young Investigator Award from MDPI, and Joukowsky Family Foundation Outstanding Dissertation Award of Brown University.
Shuang Li (Google DeepMind)
Title: From Learned Skills to Agentic Robots
Abstract: How can robots combine reliable physical skills with the ability to reason, adapt, and pursue goals over time? In this talk, I will first present the Unified Video Action Model (UVA), which jointly learns video and action generation, using video supervision to improve policy learning while enabling fast action inference. I will then discuss how embodied reasoning can guide robot actions through planning, progress monitoring, and correction of failed steps. Drawing on public examples, I will explore the opportunities and challenges this creates for agentic robotics. I will conclude with a personal perspective on whether reasoning and action could share a common foundation while operating at the different timescales required for deliberation and physical control.
Bio: Shuang Li is a Research Scientist at Google DeepMind, working on robotics and generative modeling, with a particular interest in agentic robotics. She received her Ph.D. from MIT, advised by Antonio Torralba, and completed postdoctoral research at Stanford with Shuran Song and Dorsa Sadigh. Her honors include being named one of MIT Technology Review’s 35 Innovators Under 35 and CIFAR AI Chair.