Learning Score-based (Diffusion Model)Grasping Primitive for Human-assisting Dexterous Grasping
NeurIPS 2023
Tianhao Wu12*, Mingdong Wu1*, Jiyao Zhang12, Yunchong Gan1, Hao Dong123†
* equal contributions † corresponding author
1 School of Computer Science, Center on Frontiers of Computing Studies, Peking University
2 Beijing Academy of Artificial Intelligence
3 National Key Laboratory for Multimedia Information Processing, School of Computer Science, Peking University
Abstract
The use of anthropomorphic robotic hands for assisting individuals in situations where human hands may be unavailable or unsuitable has gained significant importance. In this paper, we propose a novel task called human-assisting dexterous grasping that aims to train a policy for controlling a robotic hand's fingers to assist users in grasping objects. Unlike conventional dexterous grasping, this task presents a more complex challenge as the policy needs to adapt to diverse user intentions, in addition to the object's geometry. We address this challenge by proposing an approach consisting of two sub-modules: a hand-object-conditional grasping primitive called Grasping Gradient Field (GraspGF), and a history-conditional residual policy. GraspGF learns 'how' to grasp by estimating the gradient of a synthesised success grasping example set, while the residual policy determines 'when' and at what speed the grasping action should be executed based on the trajectory history. Experimental results demonstrate the superiority of our proposed method compared to baselines, highlighting the user-awareness and practicality in real-world applications.
Training curve of different methods Success and Posture on Success and Posture on Object Stability on seen category unseen instances and unseen category instances seen category unseen instances unseen category instances
Ours surpasses all other baselines on 4900+ objects and 200 human trajectory patterns across 5 random seeds. Ours exhibits faster convergence, achieves the highest success rate (Success: >10% improvement compare to other basleines), and generates more human-like grasp poses (Posture). Our policy tends to maintain object stability.
* Note that the for PPO, PPO(Goal) and ILAD, we only substitute the intrinsic reward with fingertip distance reward, while remain the height and success reward the same as Ours.
User-Aware Dexterous Grasping
Bowl
Vase
Bottle
Camera
Cellphone
Mug
Real World Demonstrations
Foam Brick 1
Pudding Box 1
Mug 1
Timer 1
Chips Can 1
Foam Brick 2
Pudding Box 2
Mug 2
Timer 2
Chips Can 2
Banana 1
Tomato Soup Can 1
Suagr Box 1
Cracker Box 1
Bleach Cleanser 1
Banana 2
Tomato Soup Can 2
Sugar Box 2
Cracker Box 2
Bleach Cleanser 2
Chips Can
grasp top
approach from middle
Mug
grasp edge
approach from right
Failure Cases
Thin object that is easy for robot to collide the table during grasping
Large object part that is hard to hold
Adaptation of Grasping gradient field when the hand wrist is rotating
view 1
view 2
Dynamic Human-assisting dexterous grasping