Sensor Setup & Data Pre-process

Vehicular sensor system for MS2 dataset collection

RGB stereo, NIR stereo, and GNSS/IMU sensors were installed inside the vehicle to ensure safe and reliable operation under adverse weather conditions such as rain, snow, fog, and haze. A thermal camera cannot see through a glass, so we cannot install a thermal camera inside the vehicle. Therefore, LiDAR stereo and thermal stereo were built in outside of the vehicle. LiDARs are water-proof, and thermal cameras are covered with water-proof housing.

Sensor specification

Data acquisition:

Stereo RGB, NIR, and thermal images are acquired at 15 frames per second
Stereo LiDARs data are acquired at 10 frames per second.
Navigation data of GNSS/IMU are acquired at 100 frames per second.

Synchronization:

Stereo RGB and Stereo NIR cameras are synchronized with an external synchronizer.
Stereo thermal cameras are synchronized with the sync signal of the left thermal camera.
Stereo LiDARs are synchronized using PPS outputs from the GNSS/IMU module to acquire synchronized point clouds as well as to minimize interference between LiDARs during the scanning process.

Image rectification and cropping

The original RGB, NIR, and thermal images contain unnecessary regions such as car hood and sky area.
Also, the projected LiDAR’s depth points do not appear in the sky area and are invalid in the car hood.
Therefore, after the calibration process, we rectified and cropped the original RGB, NIR, and thermal images to have valid areas only.
After rectification and cropping, RGB, NIR, and thermal images have 1224 x 384, 1280 x 352, and 640 x 256 spatial resolution, respectively.

Projected depth map generation

We conduct the following processes to generate a projected depth map for each (Left) camera coordinate system.

Estimate each sensor’s pose according to its timestamp with the navigation data (i.e., GPS/IMU).
For every single frame, merge consecutive 32 frames of stereo LiDAR scans with each sensor’s pose and refine it with the ICP method.
Project the 32-frame merged point cloud to each camera coordinate.
Filtering unreliable points with L-R consistency, prediction-GT ratio test with a fine-tuned stereo matching network, etc.

Page updated

Google Sites

Report abuse