Fulong Ma, Xiaoyang Yan, Guoyang Zhao, Xiaojie Xu, Yuxuan Liu, Jun Ma, and Ming Liu
the Hong Kong University of Science and Technology (Guangzhou), Intelligent Autonomous Driving Center (IADC)
Monocular 3D object detection plays a crucial role in autonomous driving. However, existing monocular 3D detection algorithms depend on 3D labels derived from LiDAR measurements, which are costly to acquire for new datasets and challenging to deploy in novel environments. Specifically, this study investigates the pipeline for training a monocular 3D object detection model on a diverse collection of 3D and 2D datasets. The proposed framework comprises three components: (1) a robust monocular 3D model capable of functioning across various camera settings, (2) a selective-training strategy to accommodate datasets with differing class annotations, and (3) a pseudo 3D training approach using 2D labels to enhance detection performance in scenes containing only 2D labels. With this framework, we could train models on a joint set of various open 3D/2D datasets to obtain models with significantly stronger generalization capability and enhanced performance on new dataset with only 2D labels. We conduct extensive experiments on KITTI/nuScenes/ONCE/Cityscapes/BDD100K datasets to demonstrate the scaling ability of the proposed method.
Training process of our proposed method. It shows how the pre-trained model's inference, combined with the 2D annotations from the dataset, facilitates the training of a 3D detection model on datasets that lack 3D training labels.
Training labels update process. In the left image, the pre-trained 3D detection model's predictions on new data are shown, which may include some erroneous detections, as indicated by the green boxes. The middle image illustrates the process of identifying and filtering out these erroneous detections, marking them in gray based on the matching results. The right image represents the reconstruction of the ground truth heatmap using the pseudo 3D labels.
