Embedded Files
Invisible Reflections: Leveraging Infrared Laser Reflections to Target Traffic Sign Perception
Summary
Summary
All vehicles must follow the rules that govern traffic behavior, regardless of whether the vehicles are human-driven or Connected Autonomous Vehicles (CAVs). Road signs indicate locally active rules, such as speed limits and requirements to yield or stop. Recent research has demonstrated attacks, such as adding stickers or projected colored patches to signs, that cause CAV misinterpretation, resulting in potential safety issues. Humans can see and potentially defend against these attacks. But humans can not detect what they can not observe. We have developed an effective physical-world attack that leverages the sensitivity of filterless image sensors and the properties of Infrared Laser Reflections (ILRs), which are invisible to humans. The attack is designed to affect cameras in CAV, undermining traffic sign recognition by inducing misclassification.
In this work, we formulate the threat model and requirements for an ILR-based traffic sign perception attack to succeed. We evaluate the effectiveness of the ILR attack with real-world experiments against two major traffic sign recognition architectures on four IR-sensitive cameras. Our black-box optimization methodology allows the ILR attack to achieve up to 100% attack success rates in indoor, static scenarios and >=80.5% attack success rates in our outdoor, moving vehicle scenarios. We find that the latest state-of-the-art certifiable defense is ineffective against ILR attacks as it mis-certifies >=33.5% of cases. To address this, we propose a detection strategy based on the physical properties of IR laser reflections which can detect 96% of ILR attacks.
Media Coverage
Media Coverage
Overview of Infrared Laser Reflection (ILR) Attack Vector
Overview of Infrared Laser Reflection (ILR) Attack Vector
In this work, we formulate the threat model and requirements for an ILR-based sign perception attack. We evaluate the effectiveness of the ILR attack against two major types of traffic light recognition systems on 3 IR-sensitive cameras and confirm that the ILR attack has 100% attack success rates in the indoor experiment and 90% attack success rates (ASR) in the outdoor moving experiment. We further find that the current process of state-of-the-art certifiable defense, in which classification models can keep correct predictions even if a small portion of the image is masked, does not hold for traffic sign recognition. The defense implementation makes the system even more vulnerable, with 35% of mis-certified cases.
Attack Modeling
Attack Modeling
The attacker can control the following parameters to optimize the impact of the ILR attack on traffic sign misclassification.
𝑑as - Distance: attacker ↔ sign
𝐷 - Diameter of IR pattern
(𝑥b, 𝑦b) - Coordinates of center of the IR pattern
𝑃a - Laser power
Aside from these parameters, the attack also depends on environmental factors not controllable by the attacker, such as the ambient light intensity 𝐿, the distance between the victim and the traffic sign 𝑑vs
ILR Attack Methodology
ILR Attack Methodology
We design an optimization framework based on the attack parameters to generate effective ILR attacks. The framework consists of three steps:
Image difference-based IR trace modeling: This step first empirically collects the IR patterns created by the attack for different powers of the laser beam (𝑃a) and diameters of the IR pattern (D). Since collecting traces at all powers and diameters is difficult, we interpolate the previously collected patterns to get a continuous relation between the laser power, the diameter, and the IR pattern.
Optimization-based ILR attack generation: We then design a black-box optimization formulation to optimize the attack with respect to laser power 𝑃a, IR pattern diameter D, and arbitrary trace position (𝑥b, 𝑦b). The optimization uses the interpolation method described in the previous step to synthesize IR patterns at desired power and diameter.
Attack deployment and verification on controlled attack scenarios: We set the attacker setup to the optimized power and pattern diameters and aim the IR emitter at the optimized position (𝑥b, 𝑦b) to deploy the attack.
Maximum Achievable Distance
Maximum Achievable Distance
To evaluate the maximum achievable distance from the emitter to the target traffic sign, the experiment was conducted in indoor and outdoor scenarios in a controlled environment. Using our setup, we verify the attack success rate (ASR) - meaning the percentage of misclassification cases, against a real-world speed limit sign using the LISA model.
Our results show that with our minimal setup, ILR consistently succeeds (100% ASR) up to 25 meters from the target sign with a power of 26 mW. Long-range attacks are possible because of the coherent properties of the laser beam. Beyond 25 m, speckle intensity loss and beam divergence prevent coherent pattern shape projection.
Laser Module Evaluation
Laser Module Evaluation
To demonstrate the generality of the attack, we conducted experiments with two other laser modules with 830 nm and 980 nm wavelengths. We test the attack on the Leopard Camera with the OnSemi AR023 sensor and observe a 100% ASR in all the tested two-stage detection models for both laser modules. However, the SCR for the speed limit signs was not high for other laser modules. In the stop sign case, while the 980 nm laser module showed a 100% SCR for both tested models, the 830 nm module showed 80% SCR on the GTSRB.
Outdoor Moving Experiment
Outdoor Moving Experiment
To study the ILR attack in real-world scenarios, we evaluate the attack effectiveness in an outdoor scenario with a moving vehicle. The results of this experiment demonstrate the feasibility of the ILR attack in real-world driving scenarios. In this context, we evaluate the two automotive camera sensors, the Leopard OnSemi camera, and the Leopard OmniVision camera. We record the camera image streams when the car reaches 12 meters from the sign with three speeds: 5, 8, and 13 km/h (approximately 3, 5, and 8 mph). The ASR, in this case, is calculated as the percentage of successful misclassification in terms of the number of successful frames among all the frames collected by the camera.
leoaprd1_day_speed.mp4Attack on Leopard OnSemi camera on speed limit classification during daytime.
leopard1_night_stop.mp4Attack on Leopard OnSemi camera on stop sign classification during nighttime.
leopard2_day_speed.mp4Attack on Leopard OmniVision camera on speed limit classification during daytime.
leopard2_night_stop.mp4Attack on Leopard OmniVision camera on stop sign classification during nighttime.
leopard1_day_stop.mp4Attack on Leopard OnSemi camera on stop sign classification during daytime.
leopard1_night_speed.mp4Attack on Leopard OnSemi camera on speed limit classification during nighttime.
leopard2_day_stop.mp4Attack on Leopard OmniVision camera on stop sign classification during daytime.
leopard2_night_speed.mp4Attack on Leopard OmniVision camera on speed limit classification during nighttime.
The ILR attack achieves a high ASR > 99% for the speed limit sign for all the tested speeds on ARTS and LISA. For the stop sign, on the other hand, we observe ASR > 90% for ARTS and > 80.5% in GTSRB at all speeds. The ASR for the ARTS detection model is 100% in all scenarios.
Attack Effectiveness Evaluation
Attack Effectiveness Evaluation
We selected two evaluation metrics based on our threat model and attack design: the attack success rate (ASR), which measures the percentage of cases in which a sign is misclassified or undetected, and the simulation consistency rate (SCR), which represents the percentage of cases in which the classification caused by the ILR attack is consistent between physical and digital scenarios. Along with the optimized attacks, we also evaluate random attacks, where the IR pattern is randomly projected on the target traffic sign. We evaluate all the attacks on the Leopard AR023 camera unless specifie
As shown, the ILR attacks show significantly higher attack effectiveness than the random attack, with a 100% success rate for all models. These results indicate that the IR laser traces can fool traffic sign recognition systems. Still, effective attack optimization is needed to cause a significant attack impact on the system.
Generality to Different Cameras
Generality to Different Cameras
To further study the impact of the ILR attack on other cameras, we evaluate attack effectiveness using a Raspberry Pi HQ v1.1 camera with a Sony IMX477 image sensor, a Microsoft LifeCam HD-3000 camera, and another automotive camera with an Omnivision OV10635 image sensor with the IR filter removed. The ILR attack is always successful as the ASR is 100%. On the other hand, the SCR of the speed limit sign is not high for other cameras. We observe that the color of IR traces on these cameras is succeptable to attack parameters, which may affect the simulation accuracy of the IR trace.
Attack Robustness Evaluation
Attack Robustness Evaluation
Robustness to Ambient Light: The attack is generated under 100 Lux and evaluated in other seven lighting conditions, ranging from 50 to 300 Lux. The ILR attack against the stop sign shows high robustness between 100 to 230 Lux, but the ASR suddenly drops out of the range. Meanwhile, the attack against the speed limit sign shows high robustness with 100% ASR for all ambiances. The difference in performance should be due to the difference in the surface color of the traffic sign.
Robustness to Different Camera Positions: For the two-stage architectures, the attack is generated at 2 m lateral and 1 m longitudinal positions and evaluated at all other camera positions. The lateral direction to the traffic signs has higher impacts than the longitudinal direction. As the attack is not optimized against the viewing angle, the attack performance is degraded even though we try to make the attack robust with the EoT technique. Meanwhile, the ASRs are generally high, particularly within 1 m lateral differences. For the SCR, the stop sign typically has higher values than the ones of the stop sign, while the speed limit sign has higher ASRs. For the single-stage architectures, we evaluate the robustness under four different traffic sign distances. The ILR attack for this scenario is successful with 100% ASR for all models trained on the traffic sign datasets (ARTS and Mapillary). However, the YOLOv3 model trained on the COCO dataset shows higher robustness to the ILR attack and is not vulnerable with 0% ASR when the camera potion differs.
Robustness to Different Object Detectors in Single-Stage Architecture: The ILR attack reaches high attack effectiveness for the speed limit sign with 100% ASR at all tested distances and models. For the stop sign, the ILR attack is effective against Faster R-CNN trained on the ARTS dataset but not always effective against YOLOv3 and YOLOv5. We believe these variations are due to the architectural differences in object detectors. The Faster R- CNN model, a two-shot object detector finds region proposals and classifies those regions. It thus has a high ASR similar to the second-stage classification model. YOLOv3 and YOLOv5, single-shot object detectors, perform the two steps simultaneously. This strategy may contribute to the robustness as it can take into account global features out of the region proposal. These results indicate that single-stage traffic sign recognition with a single-shot object detector can be an effective mitigation against ILR attacks. However, we note that the current object detectors are still not able to handle several types of different traffic signs.
Robustness to Inaccuracy in First-Stage Object Detection:
We also evaluate how inaccuracies in the first stage can change the automatic bounding cropping and consequently alter the input of the second-stage classification model. To evaluate the impact of the inaccuracy on the classification results, we apply vertical and horizontal translation noise to our manually annotated bounding boxes. The ASR and SCR for the stop sign decrease with increasing noise levels. In contrast, the ASR for the speed limit sign is always 100%, while the SCR eventually starts to decrease around a noise level of 8%.
Research Paper
Research Paper
[NDSS 2024] Invisible Reflections: Leveraging Infrared Laser Reflections to Target Traffic Sign Perception
Takami Sato∗, Sri Hrushikesh Varma Bhupathiraju∗ (co-first authors), Michael Clifford, Takeshi Sugawara, Qi Alfred Chen and Sara Rampazzi
To appear in the Network and Distributed System Security (NDSS) Symposium 2024
BibTex for citation:
@inproceedings{sato2024invisible,
title={{Invisible Reflections: Leveraging Infrared Laser Reflections to Target Traffic Sign Perception}},
author={Sato, Takami and Bhupathiraju, S Hrushikesh and Clifford, Michael and Sugawara, Takeshi and Chen, Qi Alfred and Rampazzi, Sara},
booktitle={{Network and Distributed System Security Symposium (NDSS)}},
year={2024}
}
Team
Team
Takami Sato, Ph.D. student, University of California, Irvine
Sri Hrushikesh Varma Bhupathiraju, Ph.D. student, University of Florida
Michael Clifford, Researcher, Toyota InfoTech Labs
Takeshi Sugawara, Assistant Professor, The University of Electro-Communications
Qi Alfred Chen, Assistant Professor, University of California, Irvine
Sara Rampazzi, Assistant Professor, University of Florida
Acknowledgments
Acknowledgments
We thank the anonymous shepherd and reviewers for their valuable comments. This research was supported in part by the NSF CNS-1932464, CNS-1929771, CNS-2145493, USDOT UTC Grant 69A3552047138, JST CREST JPMJCR23M4, and unrestricted research funds from Toyota InfoTech Labs. We want to thank Himanandhan Reddy Kottur for his help with the outdoor experiments.
Google Sites
Report abuse