Multi-fingered Robotic Hand Grasping in Cluttered Environments through Hand-object Contact Semantic Mapping
Lei Zhang, Kaixin Bai, Guowen Huang, Zhaopeng Chen, Jianwei Zhang
Abstract
The integration of optimization and generative models has significantly advanced dexterous manipulation techniques for five-fingered hand grasping. Yet, the application of these techniques in cluttered environments is a relatively unexplored area. To address this research gap, we have developed a novel method for generating five-fingered hand grasp samples in cluttered settings. This method emphasizes simulated grasp quality and the nuanced interaction between the hand and surrounding objects. A key aspect of our approach is our data generation method, capable of estimating contact spatial and semantic representations and affordance grasps based on object affordance information. Furthermore, our Contact Semantic Conditional Variational Autoencoder (CoSe-CVAE) network is adept at creating comprehensive contact maps from point clouds, incorporating both spatial and semantic data. We introduce a unique grasp detection technique that efficiently formulates mechanical hand grasp poses from these maps. Additionally, our evaluation model is designed to assess grasp quality and collision probability, significantly improving the practicality of five-fingered hand grasping in complex scenarios. Our data generation method outperform previous datasets in grasp diversity, scene diversity, modality diversity. Our grasp generation method has demonstrated remarkable success, outperforming established baselines with 81.1% success rate in real-world single-object grasping and 75.3% success rate in multi-object grasping.
Multi-modal FFH Cluttered Grasping Dataset
Contact Semantic Label
FFH Grasping Poses from Cluttered Scenes
Cluttered Scene
Grasping Poses in Cluttered Scenes
(Blue: Collision, Green: Positive Grasps, Red: Negative Grasps)
Grasp Diversity, Scene Diversity, Modality Diversity
Diversity of Grasping Scenes and Corresponding Grasping Candidates
Comparison Experiment of Grasp Optimization Process
Real World Robotic Experiments
Single-Object Grasping
Multi-Object Grasping
Failed Grasping
Dataset Links
Grasping and Manipulation Affordance
Code and example dataset: https://github.com/leizhang-public/ffh-cluttered-grasping
Grasping Dataset with Collision Scores, Grasp Validation Results, Contact Semantic and Distance Map
Code and example dataset: https://github.com/leizhang-public/ffh-cluttered-grasping
In case this work has provided some assistance in your research endeavors, your citation would be greatly appreciated.
@article{test,
title={Title},
author={authors},
journal = {TBD},
year = 2024,
url = {TBD}
}