Experiment Results on New Object Categories

Experiment Setup 

To test whether Real2Code can handle objects with complex geometries, we have collected a new dataset that included two new object categories from PartNet-Mobility - Eyeglasses and Scissors. These two categories have complex shapes beyond simpler, cuboid-like shapes (e.g. cabinets) that we evaluated on in the main paper. 

Data Preparation

To prepare the dataset, we ran mesh repairing processing to clean up the raw PartNet object assets, split them into train-test objects (52-5 for Eyeglasses, 43-4 split for Scissors, respectively), and render RGBD images using our original rendering pipeline built with blenderproc. We use the new data to re-train both our proposed SAM fine-tuning model and shape-completion model as described in the main paper, and test the new models on unseen, held-out object instances.

Experiment Results

We visualize test results on all the objects below: the input multi-view RGB images (first row), merged part-level segmentation masks from our fine-tuned SAM model (second row), and assembled shape completed meshes for each segmented object part (third row). 

Overall, we observe that 2D part segmentation is the main bottleneck of our pipeline: our model's mesh completion from GT segmentation has clearly better quality than using SAM-proposed segmentation masks, and completely fails for more extreme OOD instances where the object looks very different from training set (e.g. the glass door ID 9107).

 In contrast to the furniture-like objects, the Eyeglasses and Scissors objects posts additional challenges due to having parts with thinner shapes (e.g. blades) and contains more intricate details (e.g. handle bars),  and taking up smaller areas in the rendered RGBD images.  Qualitatively, we observe the SAM-based segmentation model is able to propose kinematically correct segmentations, but fails at some instances were the glasses legs or scissor blades are too thin.