Visualization

Clean

Weather Corruption

The sensors can obtain rich environmental information during the sunny daytime, however, we need to consider the degradation of sensor on adverse weather conditions or various lighting conditions throughout the day. Weather conditions can inevitably affect the sensor's perception, resulting in the performance degradation of MSF systems. For example, normal cameras could hardly perceive the surroundings at night. 

In this work, we leverage weather corruption patterns from two perspectives: (1) light conditions change, and (2) adverse weather conditions. 

1. Light conditions. The camera is sensitive to lighting conditions, variations in daylight and road illumination can easily affect the image quality, while lighting conditions’ effects on LiDAR are limited. Therefore, we mainly focus on adjusting the brightness (BR) and darkness (DK) of the image pixels. Specifically, we convert the color image from RGB color space to HSV space, and then change the brightness value of the pixels.

2. Weather conditions. Adverse weather can cause asymmetric measurement distortion of sensors, which poses a significant challenge for MSF perception systems that rely on redundant information.  In our benchmark, we choose the domain-specific physical model to simulate the properties of two representative adverse weather, i.e, rain (RN) and fog (FG).  

Brightness

Brightness(BR) simulates high illumination by increasing the brightness of the image.

Darkness

Similar to BR, Darkness(DK) simulates low illumination by decreasing the brightness of the image to simulate poor lighting conditions (e.g. , in the evening or early morning).

Rain

In clear weather, the atmosphere behaves like a transparent medium and transmits light with very little attenuation or scattering.  However, raining fills the atmosphere with droplets, which causes natural light or laser pulses to scatter as they pass through the atmosphere. During rainy environments, sensors could be affected by rain drops. For cameras, raindrops could lead to pixel attenuation and rain streaks on the image. For LiDAR, the droplets will make the laser scattering and absorption, resulting in a lower intensity of points and perceived quality of LiDAR.

Fog

Different from rain, fog fills the atmosphere with small particles.  These particles also make the laser scattering and absorption and degrades the visibility of a scene significantly.

Sensor Corruption

Sensor corruptions reflect internal environment changes that lead to corrupted sensor signals, e.g., noises during transmission, and sensor artifacts that lead to blurry images. In this benchmark, we consider sensor corruption from two perspectives: (1) noise pattern, and (2) sensor artifacts. 

1. Noise Pattern. Noise typically exists in both camera and LiDAR . There are two main sources of noise, one is from the sensor itself, such as sensor vibration, random reflections and the low-ranging accuracy of LiDAR lasers . The other is due to the digital signal in its transmission recording process. 

2. Sensor Artifacts. Different sensors have unique corruption patterns due to different physical properties.  Sensor corruption could also result in artifacts of sensing results. 

Image  Gaussian Noise

Image Gaussian Noise applies gaussian distributional noise to each pixel in the image.

Point Cloud Gaussian Noise

Point Cloud Gaussian Noise applies gaussian distributional noise to each point in a point cloud.

Image  Impulse Noise

Image Impulse Noise randomly changing the value of image pixels.

Point Cloud Impulse Noise

Point Cloud Impulse Noise applies applies deterministic perturbations to a subset of points.

Motion Blur

Motion Blur appears when a  camera shakes or moves rapidly, making artifacts appear on fast moving objects. In most cases, a short enough exposure time allows the camera to take an image that captures a moment. However, fast-moving objects or longer exposure times may result in blurred artifacts that become visible. 

Defocus Blur

The camera, as a typical optical device, may suffer from defocus blur. In general, defocus reduces the sharpness and contrast of the image. What should be sharp, high-contrast edges in a scene become gradual transitions. Fine detail in the scene is blurred or even becomes invisible. 

Sensor Misalignment

Well-calibrated and synchronized sensors are a prerequisite for MSF-based perception systems. However, it is not easy to guarantee the perfect alignment of sensors in the real world. MSF systems could suffer from temporal and spatial misalignment, leading to errors when fusing data from different branches. Therefore, we design two corruption patterns, Spatial misalignment (SM) and  Temporal misalignment (TM), to simulate the misalignment between the camera and LiDAR.

Recommended Parameters:  

1. SM: We set the parameters according to the daily calibration of the KITTI. KITTI recalibrates the sensors each day after their recordings to avoid sensor spatial misalignment. Specifically, KITTI calibrates angular deviations between 0.04° and 2.54° for each calibration and between 0° and 1.3° for each axis. Therefore , We design the suitable parameters (0.5°-2°) for SM corruption according to the above information. 

2. TM: There is a time desynchronization of 0.85s between the camera and LiDAR[5]. Therefore , We design the suitable parameters (0.1s-0.5s) for TM corruption according to the above information. 

3. In rare scenarios, e.g., unfixed sensors, damaged sensors, delayed data transmission, etc., the sensors may have a greater misalignment . The corruption parameters can be adjusted to evaluate the robustness of the system in corner case scenarios.

Spatial  Sensor Misalignment

MSF system requires an external calibration of each sensor during the assembly process to ensure that the position measured in different coordinate systems can be converted to each other. However, even with well-calibrated sensors, the position of the sensors will inevitably deviate due to mechanical vibrations (e.g., when a self-driving car rides on a bumpy road) and thermal fluctuations. 

 We add a slight rotation to each rotation angle (i.e., roll, yaw, pitch) to simulate spatial misalignment between the camera and LiDAR. Here is an example of how to rotate an object along a certain axis.

We show the visualization of the effects with  2° rotation around different axes by projecting the point cloud onto the image.

Temporal Sensor Misalignment

MSF system requires synchronization of sensors to ensure the output from each individual branch is sensed at the same time. In practical scenarios, sensor or transmission failure may cause a delay in one branch, resulting in a temporal misalignment. To simulate temporal misalignment, for a timestamp t, we replace the data Mi(t)with the Mi(t − Δt). This could represent a signal delay of t second on branch i. 

Signal Loss

The signal collected by the sensor may lose some information in transmission. To simulate this, for the camera branch, we reshape the image into a one-dimensional array and drop some pixels according to the percentage of signal loss. For the LiDAR branch, we randomly remove some points with different percentages.

Reference

[1] D. Hendrycks and T. Dietterich, “Benchmarking neural network ro bustness to common corruptions and perturbations,” arXiv preprint arXiv:1903.12261, 2019.

[2] V. Kilic, D. Hegde, V. Sindagi, A. B. Cooper, M. A. Foster, and V. M. Patel, “Lidar light scattering augmentation (lisa): Physics-based simulation of adverse weather conditions for 3d object detection,” arXiv preprint arXiv:2107.07004, 2021.

[3] S. S. Halder, J.-F. Lalonde, and R. d. Charette, “Physics-based rendering for improving robustness to rain,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2019, pp. 10 203–10 212.

[4] C. Sakaridis, D. Dai, and L. Van Gool, “Semantic foggy scene under standing with synthetic data,” International Journal of Computer Vision, vol. 126, no. 9, pp. 973–992, 2018.

[5] Yeong D J, Velasco-Hernandez G, Barry J, et al. Sensor and sensor fusion technology in autonomous vehicles: A review[J]. Sensors, 2021, 21(6): 2140.