Scaling All-Goals Updates in Reinforcement Learning Using Convolutional Neural Networks
Fabio Pardo, Vitaly Levdik, Petar Kormushev
AAAI 2020
Being able to reach any desired location in the environment can be a valuable asset for an agent. Learning a policy to navigate between all pairs of states individually is often not feasible. An all-goals updating algorithm uses each transition to learn Q-values towards all goals simultaneously and off-policy. However the expensive numerous updates in parallel limited the approach to small tabular cases so far. To tackle this problem we propose to use convolutional network architectures to generate Q-values and updates for a large number of goals at once. We demonstrate the accuracy and generalization qualities of the proposed method on randomly generated mazes and Sokoban puzzles. In the case of on-screen goal coordinates the resulting mapping from frames to distance-maps directly informs the agent about which places are reachable and in how many steps. As an example of application we show that replacing the random actions in epsilon-greedy exploration by several actions towards feasible goals generates better exploratory trajectories on Montezuma's Revenge and Super Mario All-Stars games.
Montezuma's Revenge room 1: exploration comparison
Random exploration
Proposed Q-map exploration
Super Mario All-Stars level 1.1: training of the proposed DQN + Q-map agent
Episode 1
Episode 12
Episode 112
Episode 1120
Super Mario All-Stars level 1.1: best episodes
DQN
proposed DQN + Q-map
Super Mario All-Stars level 2.1: training of the proposed DQN + Q-map agent
Episode 1
Episode 12
Episode 101
Episode 990
Super Mario All-Stars level 2.1: training of the proposed DQN + Q-map agent with pre-training on level 1.1
Episode 1
Episode 11
Episode 91
Episode 1245