Jeonghyun Kim, Kaichun Mo, Minhyuk Sung*, Woontack Woo*
(* Co-corresponding Authors)
WACV 2023 Algorithm Track
[arXiv] [Paper] [Supplementary] [Code] [Bibtex]
Abstract
We propose Seg&Struct, a supervised learning framework leveraging the interplay between part segmentation and structure inference and demonstrating their synergy in an integrated framework. Both part segmentation and structure inference have been extensively studied in the recent deep learning literature, while the supervisions used for each task have not been fully exploited to assist the other task. Namely, structure inference has been typically conducted with an autoencoder that does not leverage the point-to-part associations. Also, segmentation has been mostly performed without structural priors that tell the plausibility of the output segments. We present how these two tasks can be best combined while fully utilizing supervision to improve performance. Our framework first decomposes a raw input shape into part segments using an off-the-shelf algorithm, whose outputs are then mapped to nodes in a part hierarchy, establishing point-to-part associations. Following this, ours predicts the structural information, e.g., part bounding boxes and part relationships. Lastly, the segmentation is rectified by examining the confusion of part boundaries using the structure-based part features. Our experimental results based on StructureNet and PartNet demonstrate that the interplay between the two tasks results in remarkable improvements in both tasks: 27.91% in structure inference and 0.5% in segmentation.
Explanatory Video (4 min) & Poster
segstruct-post.pdfThe Interplay between Part Segmentation and Structure Inference
In the following, we illustrate the visualization of our proposed interplay between part segmentation and structure inference to build a synergy between two tasks, enhancing the performance of both tasks. Our framework fully exploits two different supervisions, such as point-to-part associations and hierarchical part structure, and thus improve performance in both tasks making a loop between them.
Figure 1. In the structure inference stage, our method predicts the plausible structures for all categories even for the cases of complex input geometry, resembling the target structures while fully utilizing part segmentation priors and point-to-part associations. The following structure-to-segmentation refinement process demonstrates how the merge operation gives us refined part segmentation by relaxing the confusion of part boundaries (red arrow). Moreover, we observe this interplay further improves the quality of part structure again.
Application: Structure-Aware Shape Retrieval
As an application based on our proposed framework, we propose a structure-aware shape retrieval, measuring a structure difference that reflects the similarity of semantics between the query shape and shapes in the shape collection. We showcase the results of top-1 shape retrieval, comparing our structure-aware retrieval with the CD-based method below.
Figure 2. The Top-1 Retrieval results of Structure-aware vs. Chamfer Distance(CD)-based approaches.
Acknowledgement
This work was supported in part by the NRF grant (No. 2021R1A2C2011459) and NST grant (No. CRC 21011) funded by the Korea government(MSIT). Minhyuk Sung acknowledges the support of grants from Adobe, KT, Samsung Electronics, and ETRI.