Discovering Agent-Centric Latent Dynamics in Theory and in Practice
ICML 2023 Tutorial
https://icml.cc/virtual/2023/tutorial/21561
John Langford, Alex Lamb
Agent-Centric Latent Dynamics
A latent state can enable vastly better planning, exploration, and credit assignment by keeping task-relevant information while discarding distractions and irrelevant details. For example, in video games, there is a game-engine state which has all relevant information for the underlying dynamics. This tutorial will discuss how we can discover such a latent state in the real world directly from observations, and the kinds of latent states which are known to be discoverable. The tutorial discusses theoretical developments at a high-level, to explain the key pieces of understanding as well as their limitations. The tutorial will discuss where the state-of-the-art is experimentally, and what is currently ready for usage in real-world applications.
ICML 2023 Presentation
Time: July 24th, 4-6pm Hawaii Time.
PDF: https://drive.google.com/file/d/1r1cnsHZZNWrnwACcKzwMcn-P4-pNwLfv/view?usp=sharing
AgentCentricTutorial_ICML2023.pdfLinks to Relevant Code
AC-State: https://github.com/alexmlamb/ControllableLatentState
ACRO (Offline-RL): https://github.com/manantomar/acro-representations
Intrepid (PPE, Homer, and more coming): https://github.com/microsoft/Intrepid
Relevant Papers
Alex Lamb, Riashat Islam, Yonathan Efroni, Aniket Didolkar, Dipendra Misra,
Dylan Foster, Lekan Molu, Rajan Chari, Akshay Krishnamurthy, and John Langford.
2022. Guaranteed discovery of controllable latent states with multi-step inverse
models. Transactions on Machine Learning Research (TMLR) 2022.
Riashat Islam, Manan Tomar, Alex Lamb, Yonathan Efroni, Hongyu Zang, Aniket
Didolkar, Dipendra Misra, Xin Li, Harm van Seijen, Remi Tachet des Combes,
et al. 2022. Agent-Controller Representations: Principled Offline RL with Rich
Exogenous Information. arXiv preprint arXiv:2211.00164 (2022).
Simon Du, Akshay Krishnamurthy, Nan Jiang, Alekh Agarwal, Miroslav Dudik,
and John Langford. 2019. Provably efficient RL with Rich Observations via Latent
State Decoding. In Proceedings of the 36th International Conference on Machine
Learning (Proceedings of Machine Learning Research, Vol. 97), Kamalika Chaudhuri
and Ruslan Salakhutdinov (Eds.). PMLR, 1665–1674. https://proceedings.mlr.press/
v97/du19b.html
Yonathan Efroni, Dylan J Foster, Dipendra Misra, Akshay Krishnamurthy, and
John Langford. 2022. Sample-Efficient Reinforcement Learning in the Presence of
Exogenous Information. arXiv preprint arXiv:2206.04282 (2022).
Yonathan Efroni, Dipendra Misra, Akshay Krishnamurthy, Alekh Agarwal, and
John Langford. 2021. Provable RL with Exogenous Distractors via Multistep
Inverse Dynamics. arXiv preprint arXiv:2110.08847 (2021).
Dipendra Misra, Mikael Henaff, Akshay Krishnamurthy, and John Langford. 2020.
Kinematic State Abstraction and Provably Efficient Rich-Observation Reinforcement
Learning. In Proceedings of the 37th International Conference on Machine
Learning (Proceedings of Machine Learning Research, Vol. 119), Hal Daumé III and
Aarti Singh (Eds.). PMLR, 6961–6971. https://proceedings.mlr.press/v119/misra20a.
html
Dipendra Misra, Qinghua Liu, Chi Jin, and John Langford. 2021. Provable Rich
Observation Reinforcement Learning with Combinatorial Latent States. In International
Conference on Learning Representations. https://openreview.net/forum?
id=hx1IXFHAw7R