1. Development of Visual Mapping Pipeline for Large Indoor Spaces (NAVER LABS; 2020-2021)

Patent : Y. Lee, S. Yeon, S. Ryu, D. Kim, and D. Lee, "Method and Systems of Generating 3D Map," U.S. Patent 17,684,763, filed March 2, 2022. Patent Pending.

Publication : D. Lee(*), S. Ryu(*), S. Yeon(*), Y. Lee(*), D. Kim, C. Han, Y. Cabon, P. Weinzaepfel, N. Guerin, G. Csurka, and M. Humenberger, "Large-scale Localization Datasets in Crowded Indoor Spaces", CVPR 2021. (* : Joint Fisrt Authors, My Independent Role : Whole "Vision Based Mapping" Part)

Motivation

• Large-scale indoor image datasets are vital to develop high-performing visual localization algorithms, however, conducting SFM on large indoor spaces are challenging due to repetitive or non-salient textures [5, 6].

• Many previous indoor image datasets are often aided by depth sensors such as RGB-D cameras or lidars, which limit their metric sizes into room-scale [2, 3, 4] (due to the range limit of RGB-D cameras) or variety of viewpoints [5] (due to difficulty of image registration into lidar scans).

Contribution

• Propose a novel fully automated mapping pipeline which jointly utilizes LiDAR SLAM and SFM.

• Model a novel SFM optimization which incorporates prior trajectory information of LiDAR SLAM.

• Introduce five new large indoor datasets of a subway and a department store. The proposed datasets are highly accurate, large-scale and densely-sampled.

2. Digital Twin Project with National Museum of Korea (NAVER LABS; 2021-2023)

Project Summary

• This collaborative project between NAVER LABS and National Museum of Korea is a long term project which aims to establish digital assets of Korean cultural heritages.

Role and Contribution

• My role was exploring the possibillity of constructing digital twin from scanned data. I studied and tested several algorithms regarding dense map generation [7, 8, 10, 12].

• On top of the visual mapping system (SFM/MVS) developed earlier, I tailored a dense mapping pipeline jointly utilizing both LiDAR PCD and MVS PCD.

• I succeeded in photo-realistically reconstructing the National Museum of Korea and other remarkable sites in South Korea (below figures show the models that I reconstructed).

Future Research Interests

• I found that traditional model based approaches perform well with carefully collected real-world data. However, it is not easy to gather complete data from every site.

• I am interested in the way to resolve the problem of incomplete data (missing and outlier contaminated) ; one direction could be fusing local geometric/semantic hints into dense mapping algorithms [8, 9], and another direction could be tightly integrating lidar point clouds into MVS to incrementally complete RGBD images. I am also interested in incorporating network based methods like NeRF [13, 14], although they do not scale well to large spaces compared to traditional methods.

3. Development of Multi-Sensor Odometry for NAVER's Autonomous Service Robot, AROUND (NAVER LABS; 2020-2023)

Project Summary

• Development of AROUND is a part of 1784 Project of NAVER, which aims to construct NAVER's 2nd headquater as a world's first robot-friendly building.

• AROUND is now a fully automated mobile robot which supports indoor delivery services in the 28 story building.

Role and Contribution

• I was responsible for modeling and deployment of service-level odometry system for AROUND, which have to be very accurate and reliable.

• I designed tightly-fused Wheel-Visual-Inertial Odometry based on the factor-graph optimization framework [16, 17].

• My multi-sensor odometry is deployed into 40 AROUND robots and providing delivery services in the building for every workday.

Future Research Interests

• I encountered a ton of abnormal cases of sensor data during the long-term deployment of service robots. I believe that it is really important to develop theoretically robust and certifiable [19] state estimation algorithm to assure the safety of robots in real-life.

• I am interested in interchange between robots and high-level maps (semantic, photo-realistic), which should not be separated. High-level maps often require an abundance of various data to be up-to-date and robots not only need localization and motion estimation, but also semantic and structural information to navigate safely in complex real environments.

4. GPS-VINS Algorithm for Indoor-Outdoor Transitional Flight of UAV (2017-2019, Master Thesis)

Awards

• Y. Lee, J. Kang, and D. Lee, "Tight fusion of GPS-VIO for Indoor-Outdoor Transitional Flight of UAV", IROS Workshop on Challenges In Vision-Based Drones Navigation, 2019. (Best Paper Candidate)

• Outstanding Master Thesis Presentation Award, 2019.

Motivation

• GPS-INS [22, 23] and VINS [16, 18] are the two major modes of state estimation for unmanned aerial vehicle (UAV), however, GPS is severely degraded near or inside buildings and VINS are not reliable outside due to abrupt illumination changes and limited textures.

• A suitable state estimation algorithm that utilizes both GPS and vision is in demand since practical applications of UAV often require smooth flights in whole indoor-outdoor spaces.

Contribution

• Model a tightly-fused GPS-VINS odometry algorithm which utilizes all the sensor information of GPS-INS and VINS (GPS, vision, baro-altmeter, magnetometer, and IMU).

• Propose a novel GPS factor model that incorporates low frequency GPS drift by a random walk bias model in the factor graph framework. 

• Show experimental results of smooth indoor-outdoor transitional flights of UAV under severe GPS degradation near a building.

Summary Slides of modelling of the proposed GPS-VINS algorithm

References

[1] J. L. Schonberger and J.M. Frahm, "Structure-from-Motion Revisited," CVPR 2016.

[2] A. Chang, A. Dai, T. Funkhouser, M. Halber, M. Niebner, M. Savva, S. Song, A. Zeng, and Y. Zhang, "Matterport3D: Learning from RGB-D data in indoor environments," 3DV 2017.

[3] A. Dai, A. X. Chang, M. Savva, M. Halber, T. Funkhouser, and M. Niebner, "Scannet: Richly-annotated 3D reconstructions of indoor spaces," CVPR 2017.

[4] H. Taira, M. Okutomi, T. Sattler, M. Cimpoi, M. Pollefeys, J. Sivic, T. Pajdla, and A. Torri, "InLoc: Indoor visual localization with dense matching and view synthesis," CVPR 2018.

[5] X. Sun, Y. Xie, P. Luo, and L. Wang, "A Dataset for Benchmarking Image-Based Localization," CVPR 2017.

[6] T. Sattler, S. Hilsenbeck, and D. Cremers, "Image-based localization using LSTMs for structured feature correlation," ICCV 2017.

[7] J. L. Schonberger, E. Zheng, M. Pollefeys, and J. M. Frahm, "Pixelwise View Selection for Unstructured Multi-View Stereo," ECCV 2016.

[8] Q. Xu and W. Tao, "Planar Prior Assisted PatchMatch Multi-View Stereo," AAAI 2020.

[9] Q. Xu and W. Tao, "Multi-Scale Geometric Consistency Guided Multi-View Stereo," CVPR 2019.

[10] M. Klingensmith, I. Dryanovski, S. S. Srinivasa, and J. Xiao, "CHISEL: Real Time Large Scale 3D Reconstruction Onboard a Mobile Device using Spatially-Hashed Signed Distance Fields," RSS 2015.

[11] M. Kazhdan, M. Bolitho, and H. Hoppe, "Poisson Surface Reconstruction," Eurographics Symposium on Geometry Processing, 2006.

[12] M. Kazhdan and H. Hoppe, "Screened Poisson Surface Reconstruction," ACM Transactions on Graphics, 2013.

[13] B. Mildenhall, P. P. Srinvasan, M. Tancik, J. T. Barron, R. Ramamoorthi, and R. Ng, "NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis," ECCV 2020.

[14] R. M. Brualla, N. Radwan, M. S. M. Sajjadi, J. T. Barron, A. Dosovitskiy, and D. Duckworth, "NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collections," CVPR 2021.

[15] K. J. Wu, C. X. Guo, G. Georgiou, and S. I. Roumeliotis, "VINS on wheels," ICRA 2017.

[16] T. Qin, P. Li, and S. Shen, "VINS-Mono: A Robust and Versatile Monocular Visual-Inertial State Estimator," IEEE Transactions on Robotics, 2018.

[17] C. Forster, L. Carlone, F. Dellaert, and D. Scaramuzza, "On-Manifold Preintegration for Real-Time Visual-Inertial Odometry," IEEE Transactions on Robotics, 2017.

[18] A. I. Mourikis and S.I. Roumeliotis, "A Multi-State Constraint Kalman Filter for Vision-aided Inertial Navigation," ICRA 2007.

[19] H. Yang, J. Shi, and L. Carlone, "TEASER: Fast and Certifiable Point Cloud Registration," IEEE Transactions on Robotics, 2020.

[20] J. Sola, "Quaternion Kinematics for the error-state Kalman filter," arXiv, 2017.

[21] L. Meier, D. Honeger, and M. Pollefeys, "PX4: A node-based multithreaded open source robotics framework for deeply embedded platforms," ICRA 2015.

[22] M. Barczyk and A. F. Lynch, "Integration of a Triaxial Magnetometer into a Helicopter UAV GPS-Aided INS," IEEE Transactions on Aerospace and Electronic Systems, 2012.

[23] J. Wendel, O. Meister, C. Schlaile, and G. F. Trommer, "An integrated gps/mems-imu navigation system for an autonomous helicopter," Aerospace Science and Technology, 2006.