Our full paper with appendix is available at arxiv.org/abs/2406.10285.

1 Introduction

We propose NutNet, a comprehensive defense that effectively defends against both Hiding Attacks and Appearing Attacks without incurring significant computational overhead. NutNet can effectively locate and mitigate the adversarial patches in the image and can be integrated as a data-preprocessing module into any pre-trained detector without requiring modifications to the original architecture

We conduct extensive experiments in both the digital and physical world to validate the effectiveness of NutNet in defending six detection models, i.e., one-stage detectors (YOLO V2-V4, SSD), two-stage detectors (Faster RCNN) and transformer-based detectors (DETR).

2 Demo of Defense

It is recommended to manually set the video resolution to 1080P for a better viewing experience.

2.1 Targeted Hiding Attack

The targeted Hiding Attack patches the target object to make it disappear from the detector. 

The video on the left shows the detection results without any defense, where we can see the person sometimes cannot be detected by the detector (with no bounding boxes).

While the video on the right shows the detection results with NutNet applied, where we can see the person can always be detected by the detector (with bounding boxes).

2.2 Untargeted Hiding Attack

The untargeted Hiding Attack uses patches to make all the objects disappear from the detector. 

The video on the left shows the detection results without any defense, where we can see the person and the chair sometimes cannot be detected by the detector (with no bounding boxes). Note that the patch being recognized as a zebra is an extra effect, the main objective of the patch is to hide something.

While the video on the right shows the detection results with NutNet applied, where we can see the person and the chair can always be detected by the detector (with bounding boxes).

2.2 Appearing Attack

The Appearing Attack (AA) fools the detector into misclassifying the patch as a specific object. The video may have slight shaking, but the robustness of the patch is strong enough to ensure that it is detected by the detector most of the time.

The video on the left shows the detection results without any defense, where we can see the patches are recognized as a person and a stop sign.

While the video on the right shows the detection results with NutNet applied, where we can see the two patches can hardly be recognized by the detector.

2.3 Other objects

We attempted to defend against targeted HA  for other objects, and NutNet demonstrated excellent defense effectiveness in all cases. 

The top images show the detection results of the object detector on the original images, while the bottom images show the detection results of the object detector on the images after applying NutNet. 

3 More Examples for Comparison

Here we provide more examples of the defensive performance of SAC (column 2), Jedi (column 3) and NutNet (column 4). The first column shows the original patched images. Note that SAC uses black masks while Jedi and NutNet use gray ones.