Multi-agent reinforcement learning (MARL) is an important and fundamental topic within agent-based research. After giving successful tutorials on this topic at EASSS 2004 (the European Agent Systems Summer School), ECML 2005, ICML 2006, EWRL 2008 and AAMAS 2009-2012, with different collaborators, we now propose a revised and updated tutorial, covering both theoretical as well as practical aspects of MARL.
Participants will be taught the basics of single-agent reinforcement learning and the associated theoretical convergence guarantees related to Markov Decision Processes. We will then outline why these convergence guarantees no longer hold in a setting where multiple agents learn. We will explain practical approaches on how to scale single agent reinforcement learning to these situations where multiple agents influence each other and introduce a framework, based on game theory and evolutionary game theory, that allows thorough analysis of the dynamics of multi-agent learning. Several research applications of MARL will be outlined in detail. The tutorial will include a practical hands-on session, where participants can experience the viability of reinforcement learning in several key application domains. Finally, a broad view on the challenges and prospects of multi-agent learning will be given.
Multi-agent systems are receiving increasing attention by the research community. Their inherent complexity makes is hard if not impossible to control such systems by design, which explains a keen interest of the community in adaptive multi-agent systems, i.e., multi-agent learning. Indeed, within the topic of multi-agent systems about 40% of the accepted full papers at AAMAS 2010 concern adaptive multi-agent systems. The tutorial aims at researchers that are faced with a multi-agent setting and consider to devise an adaptive solution. The tutorial assumes no prior knowledge specific to MARL or (evolutionary) game theory. Morning 2 and Afternoon 1 and 2 build on the fundamental concepts that are covered in Morning 1. Other than that, these three blocks are self-contained but complementary to each other.
Reinforcement learning is one of the most popular approaches to single-agent learning, because it is explicitly agent-centric, it is founded in psychological models and it provides convergence guarantees under the proper assumptions. It has been applied to multi-agent settings with promising results. However, the theoretical convergence guarantees based on classical proofs are lost, since common assumptions such as a Markovian environment are violated. A new perspective and theoretical framework for analyzing and improving MARL, including convergence guarantees, are presented in the proposed tutorial. Participants receive the necessary knowledge to apply the analysis to their specific multi-agent setting in order to devise and refine solutions based on MARL. Moreover, the MARL theory and practice presented are placed into the broader context of multi-agent learning, outlining challenges and prospects of the field.