Visual-Inertial Sensor Fusion

Project Overview

For an autonomous robot to complete any useful task, the first questions it must be able to answer are "where am I?", "where am I going?", and "what's around me?". I look at solving these localization, navigation, and obstacle avoidance tasks through the combination of visual and inertial sensors, which provide complementary information. Inertial sensors measure 3-axis translational acceleration and 3-axis rotational velocity, and a monocular camera lets us measure relative translation up to scale, and relative rotation. The combination of these sensors significantly reduces the drift incurred by simply integrating the inertial measurements and significantly improves the robustness of the vision system to aggressive motion and poor image quality.

Our work in this area focuses on the modeling choices [2] and inference schemes [1] necessary to enable high-quality pose estimation and scene reconstruction in challenging dynamic environments, while maintaining real-time performance on commodity hardware. 
Below are several sample video results of our system: 
  • VINS Extended Demo Reel and ICRA 2015 Video Results
    Video results associated with our recent paper. See this video for results in environments with many specular highlights, unstructured natural environments with many occlusions, and dynamic environments with many moving people. Final position errors are between 0.1% and 0.4% of the total path length. Extended Demo Reel includes comparisons to Google Tango on data collected using the Tango Tablet.
  • VINS Driving Demo
    Results of our system on a 2km driving loop collected using the Google Tango Tablet device. Final position error is ~0.3% of the total path length.
  • CVPR 2014 Live Demonstration
    Highlights reel accompanying a live demo presentation at CVPR14. See this video for results including walking gaits, aggressive hand-held motion, and extended mapping of indoor environments.
  • Active Search with an ARDrone 2.0
    Demonstration of active search with a micro air vehicle based on visual-inertial navigation and multi-scale object recognition.

VINS Extended Demo Reel

VINS Driving Demo


ICRA 2015 Video Results

CVPR 2014 Demonstration

Active Search with an ARDrone 2.0


Related References

  1. K. Tsotsos, A. Chiuso, and S. Soatto. Robust Inference for Visual-Inertial Sensor Fusion. In Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), May 2015 (pdfproject page)
  2. J. Hernandez, K. Tsotsos, and S. Soatto, "Observability, Identifiability, and Sensitivity of Vision-Aided Inertial Navigation", In Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), (winner, Best Conference Paper Award), May 2015 (pdfproject pageextended technical report)
  3. K. Tsotsos, A. Pretto, and S. Soatto. Visual-Inertial Ego-Motion Estimation for Humanoid Platforms. IEEE-RAS International Conference on Humanoid Robots, December 2012 (pdf, video)