This tutorial took place on May 10, 2016 at the AAMAS conference.

Introduction

Reinforcement learning (RL) is an important and fundamental topic within agent-based research, both in a single-agent setting, as well as in multi-agent domains (MARL). After giving successful tutorials on this topic at EASSS 2004 (the European Agent Systems Summer School), ECML 2005, ICML 2006, EWRL 2008, AAMAS 2009-2013, and ECAI 2013, with different collaborators, we now propose a revised and updated tutorial, covering both theoretical as well as practical aspects of single-agent and multi-agent RL.

Participants will be taught the basics of single-agent reinforcement learning and the associated theoretical convergence guarantees related to Markov decision processes. We will discuss how an agent's learning can be improved by incorporating prior knowledge through reward shaping, and how to extend the RL framework to handle multiple objectives simultaneously. We then move from single-agent to multi-agent RL, and outline why convergence guarantees no longer hold in a setting where multiple agents learn. We will explain practical approaches on how to scale single agent reinforcement learning to these situations where multiple agents influence each other and introduce a framework, based on game theory and evolutionary game theory, that allows thorough analysis of the dynamics of multi-agent learning. Several research applications of MARL will be outlined in detail. The tutorial will include a practical hands-on session, where participants can experience the viability of reinforcement learning in several key application domains.

Impact and target audience

Reinforcement learning is one of the most popular approaches to single-agent learning, because it is explicitly agent-centric, it is founded in psychological models and it provides convergence guarantees under the proper assumptions. It has been applied to multi-agent settings with promising results. However, the theoretical convergence guarantees based on classical proofs are lost, since common assumptions such as a Markovian environment are violated. A new perspective and theoretical framework for analyzing and improving MARL, including convergence guarantees, are presented in the proposed tutorial. Participants receive the necessary knowledge to apply the analysis to their specific multi-agent setting in order to devise and refine solutions based on MARL.

Both single- and multi-agent learning have received considerable interest within the community. This can be observed from the fact that each recent AAMAS conference has included one or several learning sessions. Moreover, many submissions to the yearly Adaptive and Learning Agents workshop (ALA) at AAMAS deal with reinforcement learning. This tutorial is aimed at researchers who are faced with a single- or multi-agent setting and consider to devise an adaptive solution for which learning is crucial. Moreover, it is of interest to researchers who already apply RL as we provide an up-to-date overview on recent advances in the field. The tutorial assumes no prior knowledge specific to RL or (evolutionary) game theory. The afternoon session builds on the fundamentals of single-agent RL outlined in the morning, but is otherwise self-contained and complementary.

Organized by

smARTlab

http://como.vub.ac.be/