Securing the Perception of Advanced Driving Assistance Systems Against Digital Epileptic Seizures Resulting from Emergency Vehicle Lighting

An object detector is a computer vision model designed to identify and locate specific objects within an image or video by drawing bounding boxes around them and providing a classification of their types (e.g., a car, a road sign, a pedestrian) and associated confidence score.

 
A confidence score represents the model's certainty about the presence of a specific object in a detected region, usually expressed as a probability between 0 (low certainty) and 1(high certainty).

A pre-defined threshold (0-1) is used to determine whether the detected object should be considered by the system as an object or not based on the confidence score.

The EpileptiCar phenomenon refers to the fluctuation in an object detector's confidence score when exposed to activated emergency vehicle lighting. This fluctuation can lead to significant drops in confidence (below the predefined detection threshold), potentially causing the object detector to fail to identify the emergency vehicle entirely. 

The EpileptiCar phenomenon occurs due to the intense, flashing light emitted from emergency vehicle lighting. This light alters the tonal distribution of the vehicle's colors in the camera frame, causing the object detector to misinterpret the visual data and resulting in fluctuations in its confidence score over time (affected by the flasher pattern).

No, the EpileptiCar phenomenon has been observed across a variety of commercial Advanced Driver-Assistance Systems (ADAS) and popular open-source object detectors, including YOLO, SSD, RetinaNet, and Faster R-CNN.
This issue generalizes between various state-of-the-art object detectors and commercial ADAS.

Several factors can influence the severity of the EpileptiCar phenomenon, including:

• Ambient light conditions: The phenomenon is more pronounced in low-light conditions, where the flashing lights have a greater impact on the camera's perception.

• Emergency flasher patterns: Different flashing patterns can result in varying levels of confidence score fluctuation.

• Camera settings: Shutter speed and ISO settings can impact the camera's sensitivity to the flashing lights.

• Distance to the emergency vehicle: The effect is most pronounced at certain distances, where the flasher saturates the image without obscuring the vehicle entirely.

Unfortunately, object trackers, which are often employed in ADAS to track the presence of multiple objects, have not been successful in mitigating the detection loss caused by the EpileptiCar phenomenon. 

Caracetamol is the combination of the words "Car" and "Paracetamol" (a medicine used against headaches). 

Caracetamol is a software framework designed to enhance the resilience of object detectors against the EpileptiCar phenomenon. It utilizes a multi-layer architecture, including:

1) a classifier to identify the presence of flashing lights 

2) a layer of refined object detectors trained on augmented data containing emergency vehicles with activated flashers. 

3) a layer that combines the outputs of these refined detectors with the original object detector.

Caracetamol significantly improves detection accuracy and reduces confidence score fluctuations.

Yes. 

We disclosed our findings with NHTSA, Tesla and the five manfuctorers of the ADAS analyzed in our study. 

We hope that the publication of the study will reveal our findings to the rest of the automotive and ADAS industry.

Mostly the car industry.
However, any company that designs a robot whose perception is based on video may gain insights from our findings.

We do not know.

Tesla vehicles fuse data obtained by additional sensors (e.g., RADARs and ultrasonic sensors) beyond the video cameras. Our work only focused on video cameras.

Moreover, lacking access to the specific type of object detectors and video cameras deployed in Tesla, we perform our analysis based on commonly used object detectors with other commercial ADASs available on Amazon. For this reason, there may be a discrepancy between our findings and the actual technological factors underlying the documented accidents of Tesla. 

Therefore, this question should be referred to Tesla or the NHTSA.

We hope that this paper will encourage the automotive industry to validate our findings on their ADASs and semi-autonomous cars with their object detectors.

Again, we do not know. 

We lack access to the data.

This question should be referred to Tesla or the NHTSA.

Maybe the following link can give some hints. 

It can be both, depending on whether the emergency flasher has intentionally (security issue) or unintentionally (safety) placed on the scene before it affected the car. 

Therefore, it is crucial to handle it. 

No. We published a study on split-second phantom attacks that demonstrated time-domain adversarial attacks against Tesla Model X a few years ago.
We demonstrated how attackers can trigger Tesla's autopilot to stop in the middle of the road in response to a stop sign that has been embedded by attackers into a digital advertisement for a few milliseconds (split second).