Programme

Abstract: Model-based reinforcement learning holds the promise of enabling data-efficient reinforcement learning in the real-world. However, on real-world systems we face additional challenges such as safety constraints, scarce data, and inaccurate models. In this talk, we discuss how these challenges interact, identify missing pieces towards safe model-based RL, and take first steps towards addressing them.

Bio: Felix Berkenkamp is a research scientist at the Bosch Center for Artificial Intelligence, where he leads the reinforcement learning activity. His main interest lies in the fundamental problems behind data-efficient and safe reinforcement learning required to enable learning on real-world systems. Previously, he was the workflow co-chair for ICML 2018 and received the ELLIS PhD award (2020) for his PhD thesis at ETH Zurich, where he worked together with Andreas Krause and Angella Schoellig. During this time, he held an AI fellowship from the Open Philanthropy Project, was an associated fellow at the Max Planck ETH Center for Learning Systems and a postgraduate affiliate at the Vector institute.

Abstract: Modern software and cyberphysical systems face open-ended tasks in complex environments, rendering accountability in the event of harm or injury an ever-growing challenge for both social and technical processes. Although well-understood techniques can judge whether programs obey formal properties, the real-world assurance that this process provides depends on its scope and precision. Harms can even occur when every agent operates correctly according to its model of the system and knowledge of its state. An understanding of the contribution of autonomous agents to a harm is necessary in order to consider counterfactuals and verify whether those agents acted appropriately. Philosophy and law employ decision artifacts such as beliefs, desires, and intentions as the basis for such assessments, motivating an understanding of how they arise within modern software systems. In this talk we will describe how to use formal reasoning to assure the machine analogues of these decision artifacts will be faithfully recorded for accountability processes.

Bio: Ruzica Piskac is an associate professor of computer science at Yale University. Her research interests span the areas of programming languages, software verification, automated reasoning, and code synthesis. A common thread in Ruzica's research is improving software reliability and trustworthiness using formal techniques. Ruzica joined Yale in 2013 as an assistant professor and prior to that, she was an independent research group leader at the Max Planck Institute for Software Systems in Germany. In July 2019, she was named the Donna L. Dubinsky Associate Professor of Computer Science, one of the highest recognition that an untenured faculty member at Yale can receive. Ruzica has received various recognitions for research and teaching, including the Patrick Denantes Prize for her PhD thesis, a CACM Research Highlight paper, an NSF CAREER award, the Facebook Communications and Networking award, the Microsoft Research Award for the Software Engineering Innovation Foundation (SEIF), the Amazon Research Award, and the 2019 Ackerman Award for Teaching and Mentoring.

Abstract: Although Reinforcement Learning has shown promising results, there are still numerous challenges preventing its adoption in practice. In many real-world applications, safety is a paramount requirement, which is incompatible with the trial and error nature of RL.

In particular, RL agents operate in uncertain environments, which require them to explore in order to reduce the potentially very high uncertainty by gathering information. The key problem is that typical exploration strategies choose actions at random with potentially harmful consequences for the agent or its environment.

This problem has triggered research and enormous progress in the area of safe RL. Common approaches aim to render, for instance, the exploration safe with respect to particular, previously specified safety constraints. Various research communities are active in this area, such as artificial intelligence and machine learning in general, but also formal verification and control theory. Yet, these communities often work disconnected from each other and may not even be aware of each other's results.

This tutorial provides an overview of different perspectives safe RL, in particular constrained RL and shielded RL. We highlight critical issues common to any approach to safe RL and pose the following questions:

• What is the specific understanding of safety?

• Which type of prior knowledge is needed to reason about the uncertainty of an agent's environment?

• How safe is safe RL ultimately?

We critically examine and compare the answers to these questions for various approaches, and present in-depth examples of how to exploit prior knowledge to empower RL agents with safety guarantees.

Bio: Nils Jansen is a tenured assistant professor at the Institute for Computing and Information Science (iCIS) at the Radboud University, Nijmegen, The Netherlands. He received his Ph.D. with distinction from RWTH Aachen University, Germany in 2015. Prior to Radboud University, he was a postdoc and research associate at the University of Texas at Austin. His current research is on formal reasoning about safety and dependability aspects in artificial intelligence (AI). At the heart of his research is the development of concepts from formal methods and control theory to reason about uncertainty and partial information in AI systems. He holds several grants within this area, both in academic and industrial settings. He is a member of the European Lab for Learning and Intelligent Systems (ELLIS).