A Low Cost High Definition Map Building Method for Autonomous Driving

Abstract: This article addresses a low cost high definition Map Building platform with camera sensor (assisted with IMU and GPS as option) only for autonomous driving.

1. Introduction

In an autonomous driving system, a high definition map (HD Map) with accuracy up to vehicle lane level for localization is required. The popular method to build HD Map uses LiDAR scanner to observe the street environment and get 3-D point cloud and road reflectivity map, then annotate the lanes, traffic signs and lights, walkway, special lane (such as public bus/shuttle), road markings, curbs, guard fences, flyover, tunnels and even parking lots etc. However, the devices with LiDAR, GPS (differential GPS as RTK) and IMUs are expensive, and the process to get the final map will scan the same street environments many times to overcome occlusions caused by moving objects or obstacles. Further, the in-time map update (for instance, work field or temporal detour) needs LiDAR scanner fleet to get new data for map refinement.

Another way to make HD Map at low cost, uses camera plus consumer level IMU and GPS only to capture RGB image data. For example, Mobileye proposed REM (road experience management) or Road Book, applying computer vision techniques (SLAM, SfM) to extract lanes, traffic signs (including RGB light signals), curbs and road markings etc. for map annotation. Similarly, some startups like KuanDeng, Deep-Motion and Momenta at China, Lvl5, Carmera and Mapper at USA, and Atlatec at Germany are using similar techniques.

The landmark extraction task from the captured data for camera-based HD Map building is more difficult than LiDAR-based methods, let alone 3-D reconstruction. How to improve the performance of this kind of techniques is critical to spread this low cost method among the HD Map industry.

Milz et al. addressed application of deep learning in visual SLAM [1], including depth estimation, optic flow estimation, feature correspondence, bundle adjustment (BA) and visual odometry (camera pose estimation) etc. Apparently this paper omitted the importance of semantic objects as lanes and signs.

Since the HD Map building work cannot be accomplished full-automatically, part of manual editing and adjustment is required in the process. How to build a nicely interactive GUI of map building is also an important issue. If the editing work can be done by non-professional workers (staff members without expertise in computer vision, photogrammetry and machine learning) with many efficient editing functions, the business operation cost to build HD Map would be reduced a lot.

Qualcomm’s work [2] proposed an end-to-end system for low cost crowdsourced 3D mapping method for autonomous driving. The front camera with consumer level GPS/IMU collects data online, while real time traffic sign and lane detection is used for annotation with triangulation. Offline clustering and Bundle Adjustment (BA) of multiple collected data are run for optimization. Obviously it didn’t build a map editing tool in [1] to handle error in SLAM and landmark detection. Besides, only those land marks are used for triangulation and BA. Unfortunately these land marks are too sparse, which may compromise the SLAM performance, especially for the loop closure case.

In the following session, we will propose a low cost HD Map building system for autonomous driving, where a camera-based vision module assisted with consumer level IMU and GPS devices run SLAM (simultaneous localization and mapping), add semantic objects, such as lanes/walkway/markings/traffic signs/RGB lights etc., remove moving objects and obstacles such as pedestrians, vehicles (motorists as well) and cyclists. Eventually a map editing platform is designed to support 2D object addition and erasing, transfer 2D locations to 3D space based on the plane assumption, and store the map.

2. Map Building with Semantic Objects Annotation

First, the system is based on a monocular visual keyframe-based SLAM framework, such as ORB-SLAM[3], which includes key point detection, camera pose estimation and tracking, key frame extraction, loop closure and BA etc. If there is a vehicle IMU for assistance, a visual-inertial mapping (VIM) system can fuse information from camera and IMU, such as maplab[4], in which the visual-inertial (VI) optimization module can handle specifically the camera pose estimation and tracking. Under some GPS-denied environments (street with skyscrapers, underground parking lot and tunnels etc.) , camera plus IMU are nice sensor configuration. Otherwise, GPS can help VIM in camera pose tracking[5], such as initialization of the whole VI optimization or initialization of loop closure. Usually the error of GPS localization at some position can be regarded as constant.

The data collected in the SLAM system will generate key frame pool, corresponding camera poses, key points sets and their reconstructed point cloud. Then the data enters a HD map editing platform shown in Figure 1.

Figure 2.

We assume the plane is estimated by the neighboring keypoints around landmarks and the camera calibration parameters are given in advance, then the derivation is obtained as below:

A 3D point X projects to the camera image plane as 2D image point x，here X and camera center C are denoted as 3x1 vector. The camera pose matrix in the SLAM coordinate system is denoted as R，x= [u, v, 1]^T，camera intrinsic matrix is K，then the perspective projection is formulated as:

For inverse projection，the goal is to reconstruct the ray direction as

and determine X with additional information, i.e. namda. Based on plane assumption（the known 3-D plane equation is ax + by + cz + d=0，where the normal vector is n=[a b c]^T），the position of X is computed as:

2.6 Map storage

The HD map will store the visual layer for SLAM’s image keypoints and their 3D point cloud, and the semantic layer for those landmarks. The format to save these contents could be defined by map builders or referred to public standards, as Open Drive, NDS (navigation data standard) and OSM (Open Street Map) etc.

3. Summary

This article proposed a low cost HD Map building system for autonomous driving, suitable for crowdsourced map data collection. On the one hand, we extract landmarks and add them into the semantic layer of the map. On the other hand, we remove moving obstacles to clean SLAM point clouds. It is also useful for private vehicles to build a parking lot map by itself for L-4 autonomous valet parking (AVP).

Reference

1. S Milz et al，“Visual SLAM for Automated Driving: Exploring the Applications of Deep Learning”，CVPR workshop，2018

2. O Dabeer et al，“An End-to-End System for Crowdsourced 3d Maps for Autonomous Vehicles: The Mapping Component“，arXiv 1703.10193，2017

3. R. Mur-Artal, J. M. M. Montiel, and J. D. Tardos, “ORB-SLAM: a versatile and accurate monocular SLAM system,” T-RO, 2015.

4. T. Schneider et al，“maplab: An Open Framework for Research in Visual-inertial Mapping and Localization”，IEEE Robotics and Automation Letters，2018

5. Z Tao et al，“Mapping and localization using GPS, lane markings and proprioceptive sensors”，IEEE IROS，2013

6. K He et al., “Mask R-CNN”, arXiv 1703.06870, 2017

7. Xi Pan, J Shi, P Luo, X Wang, X Tang. “Spatial As Deep: Spatial CNN for Traffic Scene Understanding”, AAAI2018

8. J Redmon,A Farhadi,“YOLOv3: An Incremental Improvement”，arXiv 1804.02767，2018

9. Z Zhu et al., “Traffic-Sign Detection and Classification in the Wild”, IEEE CVPR 2016.

10. J Mueller and K Dietmayer,“Detecting Traffic Lights by Single Shot Detection”, arXiv 1805.02523, Oct. 2018

11. O Bailo et al.,“Robust Road Marking Detection and Recognition Using Density-Based Grouping and Machine Learning Techniques”, IEEE WACV, 2017

12. S Lee et al.,“VPGNet: Vanishing Point Guided Network for Lane and Road Marking Detection and Recognition”, arXiv 1710.06288, Oct. 2017

13. C Yu et al.,“DS-SLAM: A Semantic Visual SLAM towards Dynamic Environments”, IEEE IROS 2018