Neural Grasp Distance Fields for Robot Manipulation
Thomas Weng1,2 David Held2 Franziska Meier1 Mustafa Mukadam1
1Meta AI 2Carnegie Mellon University
2023 International Conference on Robotics and Automation (ICRA)
Abstract
We formulate grasp learning as a neural field and present Neural Grasp Distance Fields (NGDF). Here, the input is a 6D pose of a robot end effector and output is a distance to a continuous manifold of valid grasps for an object. In contrast to current approaches that predict a set of discrete candidate grasps, the distance-based NGDF representation is easily interpreted as a cost, and minimizing this cost produces a successful grasp pose. This grasp distance cost can be incorporated directly into a trajectory optimizer for joint optimization with other costs such as trajectory smoothness and collision avoidance. During optimization, as the various costs are balanced and minimized, the grasp target is allowed to smoothly vary, as the learned grasp field is continuous. We evaluate NGDF on joint grasp and motion planning in simulation and the real world, outperforming baselines by 63% execution success while generalizing to unseen query poses and unseen object shapes.
Videos
Short Presentation (6 min.)
Demonstration Video
Approach
Our approach is inspired by DeepSDF, which implicitly represents 3D shapes as the distance between query points and the shape surface. In our case, we represent the manifold of grasps for an object as the distance between query poses and the closest grasp.
Compared to existing approaches which output discrete grasp candidates, our grasp distance representation can be easily modeled as a cost. Minimizing the cost moves a gripper pose to a successful grasp pose on the manifold.
We can go beyond optimizing just the gripper pose, to optimizing the full robot arm trajectory. We incorporate the grasp cost from NGDF into a trajectory optimizer for joint grasp and motion planning. Our differentiable pipeline passes gradients from costs to joint angles.
During optimization, the grasp target is able to smoothly vary, and is balanced with other trajectory costs such as smoothness and collision avoidance.
Design Decisions for 6-DOF Grasp and Motion Planning Pipelines
In the paper, we summarize the most important design decisions for 6-DOF grasping and motion planning pipelines. We show that NGDF eliminates the need for a heuristic grasp selection step, since the grasp pose is jointly optimized with motion planning.
Evaluations
In simulation evaluations with a Franka arm, we find our approach outperforms existing joint grasp and motion planning methods that use discrete grasp prediction. We also demonstrate generalization to unseen poses and object shapes.
Finally, we demonstrate grasp trajectory optimization with NGDF on a real robot system. We achieve an 81% success rate on reaching and grasping 9 test objects with 27 total trials. See the Demo Video above for videos of grasp trials.
(a) Visualizing the plan and imperfect object point cloud; (b) executing the plan on hardware (cameras highlighted with red boxes); (c) lifting the object. (d) The nine objects used for testing. (e) Successful grasps.
Bibtex
@article{weng2023ngdf,
title={Neural Grasp Distance Fields for Robot Manipulation,
author={Weng, Thomas and Held, David and Meier, Franziska and Mukadam, Mustafa},
journal={IEEE International Conference on Robotics and Automation (ICRA)},
year={2023}
}
Additional Details
A previous version of this work appeared in the IROS 2022 workshop on uncertainty in manipulation.
Acknowledgements
This work was supported by the US Air Force and DARPA (FA8750-18-C-0092), NSF (IIS-1849154, DGE2140739), CMU GSA/Provost Conference Funding, and the Meta AI Mentorship Program. The authors thank Kalyan Alwala and Adithya Murali for early prototyping, as well as Taosha Fan, Austin Wang, Daniel Seita, and Chuer Pan for helpful discussions and feedback.