Background

Humans and other animals are exquisitely capable of generating and pursuing abstract goals. Reinforcement learning (RL) is a framework that is especially equipped to understand how these goals are achieved, and has become a gold standard for modeling behavior. However, most research applying RL in psychology and neuroscience rests on the assumption that subjects maximize the experimenter’s measure of performance [1]. Work from the past two decades has revealed that natural agents also have intrinsic objectives: motivations to maximize other goals that compete with maximizing task-associated rewards [2]. Indeed, many intrinsic objectives have now been proposed to study natural behavior such as novelty [3], surprise [4], learning progress [5,6], empowerment [7,8] and occupancy [9].

At the same time, research on artificial intelligence (AI) has made parallel efforts to build intrinsic motivations in artificial agents to help them interact with the world in useful ways [10,11]. Some of these intrinsic objectives for artificial agents map one-to-one to the ones used to understand natural behavior (e.g. novelty [12], surprise [13,14], learning progress [13,15], empowerment [16]). For others, the correspondence to empirical research is not trivial (e.g. compressibility [17], regularity [18]). Additionally, the intrinsic objectives developed in robotics and AI are applied in different contexts and for different purposes than the objectives identified in natural agents in psychology and neuroscience. For example, in artificial agents, intrinsic objectives are typically used to enhance exploration and are removed once a certain reward-based performance is attained. In natural agents, one interpretation casts intrinsic objectives as biases that -- while similarly aiding exploration -- are not meant to be annealed away. This distinction makes sense: natural agents are historical beings that have evolved in, and constructed, specific ecological niches that could push different types of intrinsic objectives to be more relevant than others. In the same fashion, artificial agents are designed to solve specific problems, with diverse sources of inspiration. As a consequence, we have a diverse web of intrinsic objectives, each one capturing a relevant feature of behavior, but with no principled way of figuring out how these intrinsic objectives come to be, or the extent to which they are applied by natural agents or should be applied for artificial agents. Are there any principles for intrinsic motivations? Is it possible to identify which intrinsic objectives are applicable for which contexts and agents? Are we blind people discovering distinct parts of the same elephant?

The importance of studying intrinsically-motivated, open-ended learning agents has been identified before, with a growing community meeting regularly at the IMOL workshop, an important event to work towards a better understanding of intrinsic motivation for artificial agents. In this workshop, we would particularly like to address the specific relationship between biological and artificial intrinsic objectives in the context of RL, and how that relationship can inform general principles for intrinsic motivations.

Our hope is that presenters and attendees alike leave with new ideas on how to collaboratively find principles of intrinsically motivated behavior. We believe bridging these typically separate communities can bring further understanding of the principles of intrinsic motivation.