Speaker: Zakaria Mhammedi (Google Research)
Title: Decoupling Exploration and Policy Optimization: Uncertainty-Guided Tree Search for Hard Exploration
Paper: https://arxiv.org/abs/2603.22273
Slides: TBD
More Details:
Authors: Zakaria Mhammedi, James Cohan
Abstract: The process of discovery requires active exploration -- the act of collecting new and informative data. However, efficient autonomous exploration remains a major unsolved problem. The dominant paradigm addresses this challenge by using Reinforcement Learning (RL) to train agents with intrinsic motivation, maximizing a composite objective of extrinsic and intrinsic rewards. We suggest that this approach incurs unnecessary overhead: while policy optimization is necessary for precise task execution, employing such machinery solely to expand state coverage may be inefficient. In this paper, we propose a new approach that explicitly decouples exploration from policy optimization and bypasses RL entirely during the exploration phase. Our method uses a tree-search strategy inspired by the Go-With-The-Winner algorithm, paired with a measure of uncertainty to systematically drive exploration. By removing the overhead of policy optimization, our approach explores an order of magnitude more efficiently than standard intrinsic motivation baselines on hard exploration benchmarks. Further, we demonstrate that the trajectories discovered during exploration can be distilled into deployable policies using existing supervised backward learning algorithms, achieving state-of-the-art performance by a wide margin on Montezuma's Revenge, Pitfall!, and Venture without relying on domain-specific knowledge. Finally, we demonstrate the generality of our framework in high-dimensional continuous action spaces by solving the MuJoCo Adroit dexterous manipulation and AntMaze tasks in a sparse-reward setting, directly from image observations and without expert demonstrations or offline datasets. To the best of our knowledge, this has not been achieved before for the Adroit tasks.
Speaker Bio: Zak Mhammedi is a Research Scientist at Google Research, focusing on reinforcement learning and optimization. He completed his PhD in Computer Science at the Australian National University and previously held a postdoctoral position at MIT. Zak’s work bridges the gap between theoretical and practical AI, particularly in developing efficient reinforcement learning algorithms. He has presented at top conferences such as COLT, NeurIPS, and ICML, with several papers receiving oral and spotlight recognition.
Speaker: Daniel Russo (Columbia University)
Title: Success Conditioning as Policy Improvement: The Optimization Problem Solved by Imitating Success
Paper: https://arxiv.org/abs/2601.18175
Slides: link
More Details:
Authors: Daniel Russo
Abstract: A widely used technique for improving policies is success conditioning, in which one collects trajectories, identifies those that achieve a desired outcome, and updates the policy to imitate the actions taken along successful trajectories. This principle appears under many names -- rejection sampling with SFT, goal-conditioned RL, Decision Transformers -- yet what optimization problem it solves, if any, has remained unclear. We prove that success conditioning exactly solves a trust-region optimization problem, maximizing policy improvement subject to a χ2 divergence constraint whose radius is determined automatically by the data. This yields an identity: relative policy improvement, the magnitude of policy change, and a quantity we call action-influence -- measuring how random variation in action choices affects success rates -- are exactly equal at every state. Success conditioning thus emerges as a conservative improvement operator. Exact success conditioning cannot degrade performance or induce a dangerous distribution shift, but when it fails, it does so observably, by hardly changing the policy at all. We apply our theory to the common practice of return thresholding, showing that this can amplify improvement, but at the cost of potential misalignment with the true objective.
Speaker Bio: Daniel Russo is an associate professor in the Decision, Risk, and Operations division of Columbia Business School. He completed his undergraduate studies in math and economics at the University of Michigan, doctoral studies at Stanford University under the supervision of Benjamin Van Roy, and worked as a postdoctoral researcher at Microsoft Research New England. His research has been recognized by several awards in the operations research community: the George Nicholson Prize (best paper by a PhD student), the JFIG Paper Award (best paper by a junior faculty member), the Frederick W. Lanchester Prize (best contribution to operations research in the past five years), and the Erlang Prize (early career award for contributions to applied probability). He currently serves as an associate editor at Management Science, Operations Research, and Stochastic Systems.