Abstract

Mapping and localization, preferably from a small number of observations, are two fundamental tasks in robotics. We address these tasks by combining spatial structure and end-to-end learning in a novel neural network architecture: the Differentiable Mapping Network (DMN). The DMN constructs a spatially structured view-embedding map and uses it for subsequent visual localization with a particle filter. Importantly, the DMN architecture is end-to-end differentiable, so we can jointly learn mapping and localization using gradient descent. We apply the DMN to sparse visual localization, where a robot needs to localize in a new environment with respect to a small number of images from known viewpoints. We evaluate the DMN using simulated environments and a challenging real-world Street View dataset. We find that the DMN learns effective map representations for visual localization; and the benefit of structure increases with less training data, larger environments, and more observations for mapping.

Paper

Peter Karkus, Anelia Angelova, and Rico Jonschkowski, "Differentiable Mapping Networks: Learning Structured Map Representations for Sparse Visual Localization", International Conference on Robotics and Automation (ICRA), 2020.

https://arxiv.org/abs/2005.09530

Talk

Differentiable Mapping Network

The Differentiable Mapping Network (DMN) is a novel neural network architecture that learns a spatial view-embedding map for sparse localization.

For each context input Cⁱ the DMN computes an embedding Vⁱ (top left of figure). The set of embeddings and viewpoint coordinates taken together make up the map representation m (top right), which is used for localization (bottom) using a differentiable particle filter where particles correspond to query pose candidates.

Starting with a set of particles sampled from the initial belief b₀, the observation model updates the particle weights by comparing the query image Q_t to the map using egocentric spatial attention, i.e., attention based on the relative pose of the context viewpoints and the particle. The transition model updates particles with the egomotion input ∆t. At each time step the query pose output s_t is estimated by the weighted mean of particles.

Examples