Learning to navigate in cities without a map

Piotr Mirowski, Matthew Koichi Grimes, Mateusz Malinowski, Karl Moritz Hermann, Keith Anderson, Denis Teplyashin, Karen Simonyan, Koray Kavukcuoglu, Andrew Zisserman, Raia Hadsell

DeepMind

London, United Kingdom

Navigating through unstructured environments is a basic capability of intelligent creatures, and thus is of fundamental interest in the study and development of artificial intelligence. Long-range navigation is a complex cognitive task that relies on developing an internal representation of space, grounded by recognisable landmarks and robust visual processing, that can simultaneously support continuous self-localisation (``I am here'') and a representation of the goal (``I am going there''). Building upon recent research that applies deep reinforcement learning to maze navigation problems, we present an end-to-end deep reinforcement learning approach that can be applied on a city scale. Recognising that successful navigation relies on integration of general policies with locale-specific knowledge, we propose a dual pathway architecture that allows locale-specific features to be encapsulated, while still enabling transfer to multiple cities. We present an interactive navigation environment that uses Google Street View for its photographic content and worldwide coverage, and demonstrate that our learning method allows agents to learn to navigate multiple cities and to traverse to target destinations that may be kilometres away.

In addition to our NeurIPS 2018 paper (also available on arXiv) and supplemental material, you can download the 644 lat/long locations of the landmarks used for goal representation, manually picked across London, New York City and Paris and stored as a tab-separated file.

As part of our publication, we are releasing the code for the StreetLearn environment and RL-based agent and the StreetLearn dataset, based on Google Street View images of New York and Pittsburgh. The StreetLearn dataset is a limited set of Google Street View images approved for use with the StreetLearn project and academic research.