Peter Karkus**, Mehdi Mirza*, Arthur Guez*, Andrew Jaegle*, Timothy Lillicrap*, Lars Buesing*, Nicolas Heess*, Theophane Weber*
*DeepMind, **National University of Singapore
Intelligent robots need to achieve abstract objectives using concrete, spatiotemporally complex sensory information and motor control. Tabula rasa deep reinforcement learning (RL) has tackled demanding tasks in terms of either visual, abstract, or physical reasoning, but solving these jointly remains a formidable challenge. One recent, unsolved benchmark task that integrates these challenges is Mujoban, where a robot needs to arrange 3D warehouses generated from 2D Sokoban puzzles. We explore whether integrated tasks like Mujoban can be solved by composing RL modules together in a sense-plan-act hierarchy, where modules have well-defined roles similarly to classic robot architectures. Unlike classic architectures that are typically model-based, we use only model-free modules trained with RL or supervised learning. We find that our modular RL approach dramatically outperforms the state-of-the-art monolithic RL agent on Mujoban. Further, learned modules can be reused when, e.g., using a different robot platform to solve the same task. Together our results give strong evidence for the importance of research into modular RL designs.
Tech report: https://arxiv.org/pdf/2010.01298.pdf
Short paper presented at the Beyond Tabula-Rasa in RL workshop, ICLR 2020
The Mujoban domain embeds Sokoban puzzles in a physical 3D world. The robot receives partial, first-person camera observations and has to push boxes (yellow) onto targets (red) in previously unseen configurations.
Can you solve Sokoban puzzles? Try here
We propose a modular RL architecture for embodied planning problems following a sense-plan-act hierarchy, where separate modules perform perception (“sense”), abstract reasoning (“plan”), and low-level motor control (“act”). In contrast to most hierarchical RL approaches we carefully prescribe the role of each module in the hierarchy, aiming to build in structure specific to embodied tasks, as well as knowledge about useful abstractions. This modular design allows us to choose effective network structures and training regimes, and it also allows us to reuse trained modules, e.g., for different tasks or on different robot platforms.
Our modular RL architecture consists of three modules which are trained to optimize different objectives: a controller is trained for goal-oriented locomotion and pushing; a planner is trained for high-level abstract reasoning; and a perception network is trained to infer an abstract 2D state from sequences of first-person RGB inputs. We do not impose modular structure on the abstract reasoning problem itself, but learn it only from physical interactions, for which we introduce a simple time-abstracted RL algorithm based on MPO.
Modular RL improves the state-of-the-art in Mujoban by a large margin (from 9.4% to 78.7% success with first-person only input). The videos show examples of the Modular RL agent in previously unseen Mujoban environments. The agent quickly builds good abstract representations, moves around and pushes boxes, and solves many levels that are hard even for a human player.
On the left of the screen, 'Agent input' shows the first-person camera input. Notice that it is hard to follow what the agent is doing - inferring and solving puzzles from first-person view is challenging for humans too. For reference we show a 'Top-down view' in the bottom right of the screen; however, this is not accessible to the agent. Notice that solving many of the puzzles even from the full top-down view is non-trivial.
We further show outputs of each module. 'Abstract state' depicts the output of the perception module, a prediction of the underlying Sokoban game state. 'True abstract state' shows the underlying true state, that is not accessible to the agent. Notice that in the beginning of the trajectories only parts of the abstract state prediction are correct. As the agent moves around the predictions quickly improve. 'Planned instruction' shows the high-level instruction output of our planner module.
Can we transfer and reuse RL modules similarly to modules of classic robot systems? In these experiments we replace the default 2-DoF ball robot of Mujoban with the 8-DoF Mujoco ant. We train a new controller module for the ant, but transfer the pre-trained planner module without any additional training. We use a setting where abstract states are directly observed, and thus we do not need the perception module. Despite the significantly more challenging motor control problem, success rates with the ant are only slightly lower than with the ball body (73.7% vs. 81.8%).
We compare with standard monolithic architectures (ResNet + LSTM) trained with the MPO RL algorithm. Monolithic RL can learn to solve almost none but the easiest levels.
Easy level examples:
Hard level examples: