FuseMODNet
Real-Time Camera and LiDAR based Moving Object Detection for robust low-light Autonomous Driving
Hazem Rashed, Mohamed Ramzy, Victor Vaquero, Ahmad El Sallab, Ganesh Sistu, Senthil Yogamani
Moving object detection is a critical task for autonomous vehicles. As dynamic objects represent higher collision risk than static ones, our own ego-trajectories have to be planned attending to the future states of the moving elements of the scene. Motion can be perceived using temporal information such as optical flow. Conventional optical flow computation is based on camera sensors only, which makes it prone to failure in conditions with low illumination. On the other hand, LiDAR sensors are independent of illumination, as they measure the time-of-flight of their own emitted lasers. In this work we propose a robust and real-time CNN architecture for Moving Object Detection (MOD) under low-light conditions by capturing motion information from both camera and LiDAR sensors. We demonstrate the impact of our algorithm on KITTI dataset where we simulate a low-light environment creating a novel dataset “Dark-KITTI”. We obtain a 10.1% relative improvement on Dark-KITTI, and a 4.25% improvement on standard KITTI relative to our baselines. The proposed algorithm runs at 29 fps on a standard desktop GPU using 256×1224 resolution images.
We propose an architecture where we first simulate low-illumination images using Image-to-image translation techniques. We use the images we obtained to generate optical flow which is illumination dependent as it is based on camera sensor. In addition to optical flow from cameras, we generate dense optical flow from LiDAR sensor which is illumination independent. That way we aim to capture motion information even at night.
We argue that motion signals from cameras and LiDAR are complementary to each other. Unlike LiDAR, cameras provide dense image which provides information about the far objects while LiDAR sensor provides a sparse map. Sparsity increases as the distance from the sensor increases , thus it is difficult for LiDAR to understand motion for far objects. On the other hand, due to low illumination, optical flow from camera is prone to error which is compensated from LiDAR sensor. In our empirical study, we provide results for various fusion algorithms and prove that optical from LiDAR is an added value for the system in normal day images as well as low-light scenes.
Dataset Preparation
We provide an extended version of KittiMoSeg dataset. KittiMoSeg provides 1300 images from different KITTI scenes. We extend the data using the same methodology to be 12919 images for KITTI Raw sequences. The data is publicly available for download. The provided data is the motion masks only where the corresponding RGB frames are found in KITTI website.
Dataset Samples
Below are samples of our public dataset. Weak annotation is provided using semi-automatic method and refined using manual annotation. The binary masks highlight moving objects only which correspond to the green masks on the RGB image.
Results
Below are samples of the results using our algorithm on KITTI on the left column and simulated Dark-KITTI on the right. . (a),(b) show the input RGB images. (c),(d) show rgbFlow. (e) shows lidarFlow. (f) shows Ground Truth. (g),(h) show output using RGB-only. (i),(j) show output of (RGB + rgbFlow). (k),(l) show output of (RGB + lidarFlow). (m),(n) show output of our proposed algorithm (RGB + rgbFlow + lidarFlow).
Citation
If you use this dataset in your research, please cite this publication :
@InProceedings{Rashed_2019_ICCV_Workshops,author = {Rashed, Hazem and Ramzy, Mohamed and Vaquero, Victor and El Sallab, Ahmad and Sistu, Ganesh and Yogamani, Senthil},title = {FuseMODNet: Real-Time Camera and LiDAR Based Moving Object Detection for Robust Low-Light Autonomous Driving},booktitle = {The IEEE International Conference on Computer Vision (ICCV) Workshops},month = {Oct},year = {2019}}Authors
Hazem Rashed
AI Researcher at Valeo
Mohamed Ramzy
AI Research Intern at Valeo
Victor Vaquero
PhD Student at IRI (UPC-CSIC)
Ahmad El Sallab
AI Senior Expert at Valeo
Ganesh Sistu
AI Researcher at Valeo
Senthil Yogamani
AI Senior Expert at Valeo