Abstract—We introduce Panoptic Saliency Ranking (PSR), a new task that aims to estimate the visual saliency of all objects within a scene. Unlike salient object ranking, which focuses only on ranking salient regions, PSR extends the problem to the entire scene, providing a holistic and fine-grained understanding of visual saliency. To facilitate this task, we construct PSR18K, the first large-scale benchmark for panoptic saliency ranking. It contains 17,861 images with 146,460 annotated object instances spanning 13 superclasses and 376 fine-grained categories. In addition, PSR18K provides relation graph annotations to explicitly model semantic and spatial relationships among objects. We further propose an efficient Intrinsic Relation Graph Network (IRGNet), which reinterprets Transformer self-attention as an implicit relational matrix and seamlessly transforms it into an explicit relation graph, thereby obviating the need for additional relation modeling and significantly improving inference speed. Extensive experiments on PSR18K and three widely used SOR benchmarks demonstrate that our IRGNet achieves new state-of-the-art performance and faster inference with fewer parameters. 

Proposed Dataset—PSR18K

We introduce PSR18K, the first large-scale benchmark dataset for panoptic saliency ranking. It comprises 17,861 images with 146,460 annotated instances, spanning 13 superclasses and 376 fine-grained subcategories.

Proposed Method

We propose IRGNet, an efficient framework that leverages the intrinsic relational structure of Transformer self-attention to construct explicit relation graphs, eliminating the need for heavy relation extractors and achieving SOTA performance and faster inference.

Comparison to SOTA Methods.