We have prepared 8 more test objects from Microwave and Door categories of PartNet to test zero-shot generalization ability of Real2Code, i.e these two categories were never seen in our training set. Many Microwave or Door objects have similar structural complexity to the StorageFurniture objects we trained on (e.g. a microwave also has a box-like parent body and a hinge door), but contains novel visual appearances and new geometries unseen from the training set. Therefore, our LLM module can generalize to these objects, but our fine-tuned SAM model and our shape completion model does not handle the OOD geometries well (e.g. the press buttons in a microwave dial panel). 

Below are visualizations of the input multi-view RGB images (first row), merged part-level segmentation masks from our fine-tuned SAM model (second row), and assembled shape completed meshes for each segmented object part (third row). We observe that 2D part segmentation is the main bottleneck of our pipeline: our model's mesh completion from GT segmentation has clearly better quality than using SAM-proposed segmentation masks, and completely fails for more extreme OOD instances where the object looks very different from training set (e.g. the glass door ID 9107).

Door