Learning 6-DoF Fine-grained Grasp Detection Based on Part Affordance Grounding

Yaoxian Song†, Penglei Sun†, Piaopiao Jin, Yi Ren, Yu Zheng, Zhixu Li, 

Xiaowen Chu, Yue Zhang, Tiefeng Li✉ , Jason Gu✉

Paper Code Dataset BibTeX

Robotic grasping is a fundamental ability for a robot to interact with the environment. Current methods focus on how to obtain a stable and reliable grasping pose in object wise, while little work has been studied on part (shape)-wise grasping which is related to fine-grained grasping and robotic affordance. Parts can be seen as atomic elements to compose an object, which contains rich semantic knowledge and a strong correlation with affordance. However, lacking a large part-wise 3D robotic dataset limits the development of part representation learning and downstream application. In this paper, we propose a new large Language-guided SHape grAsPing datasEt (named LangSHAPE) to learn 3D part-level affordance and grasping ability. We design a novel two-stage fine-grained robotic grasping network (named LangPartGPD), including a novel 3D part language grounding model, and a part-aware grasp pose detection model, in which explicit language input from human or LLMs could enhance the explainability of grasping decision. To evaluate the effectiveness of our proposed method, we perform fine-grained grasp detection experiments on both simulation and physical robot settings, following language instruction at multiple levels of difficulty. Results show our method achieves competitive performance in 3D geometry fine-grained grounding, object affordance inference, and 3D part-aware grasping tasks.

@article{song2023learning,

  title={Learning 6-DoF Fine-grained Grasp Detection Based on Part Affordance Grounding},

  author={Song, Yaoxian and Sun, Penglei and Ren, Yi and Zheng, Yu and Zhang, Yue},

  journal={arXiv preprint arXiv:2301.11564},

  year={2023}

}