Indoor navigation is a difficult task, as it generally comes with poor GPS access, forcing solutions to rely on other sources of information. While significant progress continues to be made in this area, deployment to production applications is still lacking, given the complexity and additional requirements of current solutions. Here, we introduce an efficient, real-time and easily deployable deep learning approach, based on visual input only, that can predict the direction towards a target from images captured by a mobile device. Our technical approach, based on a novel graph-based path generation method, combined with explainable data augmentation and curriculum learning, includes contributions that make the process of data collection, annotation and training, as automatic as possible, efficient and robust. On the practical side, we introduce a novel large-scale dataset, with video footage inside a relatively large shopping mall, in which each frame is annotated with the correct next direction towards different specific target destinations. Different from current methods, ours relies solely on vision, avoiding the need of special sensors, additional markers placed along the path, knowledge of the scene map or internet access.

Below is a systematic overview of our approach:

Visual representations and navigation demos

We combined optical flow and monocular depth and solved for camera rotation with a least-squares method. The key parameter we extract is the yaw rotation (turning left or right). The raw motion signal is noisy, so we smoothed it and converted it into 8 discrete classes. This gave us clean labels for training. We trained a lightweight convolutional neural network on image sequences. We used data augmentation, explainability-guided masking, and curriculum learning to make the model robust in crowded scenes. The trained model predicts directions in real time from the phone’s camera feed. We implemented this with the view of a mobile application with a simple compass-like interface. The final compass guidance is reliable throughout the mall even in moderately crowded areas and after scene modifications.

Below we make our entire Mall Navigation dataset and the Code publicly available 

Inside Knowledge Pipeline Codebase - link here

Mall Navigation dataset - link here