Learning Action Relations for Reinforcement Learning

Ayush Jain*¹ Norio Kosaka*² Kyung-Min Kim^2,4 Joseph J. Lim^3,4

¹University of Southern California (USC), ²NAVER CLOVA, ³KAIST, ⁴NAVER AI Lab

International Conference on Learning Representations (ICLR), 2022

Intelligent agents can solve tasks in various ways depending on their available set of actions. However, conventional reinforcement learning (RL) assumes a fixed action set. This work asserts that tasks with varying action sets require reasoning of the relations between the available actions. For instance, taking a nail-action in a repair task is meaningful only if a hammer-action is also available. To learn and utilize such action relations, we propose a novel policy architecture consisting of a graph attention network over the available actions. We show that our model makes informed action decisions by correctly attending to other related actions in both value-based and policy-based RL. Consequently, it outperforms non-relational architectures on applications where the action space often varies, such as recommender systems and physical reasoning with tools and skills.

Attention in AGILE (1) Learns tool associations in CREATE, (2) Ensures dig-skill is available before entering Lava in Grid World, (3) Extracts item statistics in RecSim