Benchmarking Object Detection Robustness against Real-World Corruptions

With the rapid recent development, deep learning-based object detection techniques have been applied to various real-world software systems, especially in safety-critical applications like autonomous driving. However, few studies are conducted to systematically investigate the robustness of state-of-the-art object detection techniques against real-world image corruptions and yet few benchmarks of object detection methods in terms of robustness are publicly available. To bridge this gap, we initiate to create a public benchmark of COCO-C and BDD100K-C, composed of sixteen real-world corruptions according to the real damages in camera sensors and potential bugs in camera's image pipeline. Based on that, we further perform a systematic empirical study and evaluation of twelve representative object detectors covering three different categories of architectures (i.e., two-stage, one-stage, transformer architectures) to identify the current challenges and explore future opportunities.

Our key findings include (1) the proposed real-world corruptions pose a threat to object detectors, especially for the corruptions involving colour changes, (2) a detector with a high mAP may still be vulnerable to real-world corruptions, (3) if there are potential cross-scenarios applications, the one-stage detectors are recommended, (4) when object detection architectures suffer from real-world corruptions, the effectiveness of existing robustness enhancement methods is limited, and (5) two-stage and one-stage object detection architectures are more likely to miss detect objects compared with transformer-based methods against the proposed corruptions. Our results highlight the need for designing robust object detection methods against real-world corruption and the need for more effective robustness enhancement methods for existing object detectors.

We make our scripts to replicate our experiments public at real-world corruption and our benchmark data at corruption dataset. Due to the due to the limitation of the storage space limit for online documents and the large size of every corruption dataset, we share three representative corruptions dataset of COCO-C. All of the COCO-C and BDD100k-c will be released in the future.