Tyler Ga Wei Lum* 1, Martin Matak* 2, Viktor Makoviychuk 3, Ankur Handa 3, Arthur Allshire 4, Tucker Hermans 3, Nathan D. Ratliff** 3, Karl Van Wyk** 3
Conference on Robot Learning (CoRL) 2024
1 Stanford University, 2 University of Utah, 3 NVIDIA, 4 University of California, Berkeley
(*), (**) indicates dual first and last author, respectively
DextrAH-G (Dexterous Arm-Hand Grasping) is a safe, continuously reacting pixels-to-action policy that achieves state-of-the-art dexterous grasping in the real world and was trained entirely in simulation using RL and teacher-student distillation with a geometric fabric controller.
A pivotal challenge in robotics is achieving fast, safe, and robust dexterous grasping across a diverse range of objects, an important goal within industrial applications. However, existing methods often have very limited speed, dexterity, and generality, along with limited or no hardware safety guarantees. In this work, we introduce DextrAH-G, a depth-based dexterous grasping policy trained entirely in simulation that combines reinforcement learning, geometric fabrics, and teacher-student distillation. We address key challenges in joint arm-hand policy learning, such as high-dimensional observation and action spaces, the sim2real gap, collision avoidance, and hardware constraints. DextrAH-G enables a 23 motor arm-hand robot to safely and continuously grasp and transport a large variety of objects at high speed using depth inputs, generalizing across object geometry.
Our proposed framework consists of three stages. First, we train a teacher privileged fabrics-guided policy (FGP) using reinforcement learning. Second, we distill the teacher FGP into a student depth FGP that is trained to imitate the teacher's actions and predict object position. Third, we deploy the depth FGP zero-shot in the real world. We leverage the same geometric fabric as an underlying action space for all of these stages.
(Left): clean depth image from the simulator. (Middle): depth image from the simulator after added noise. (Right): depth image in the real world on a novel object.
DextrAH-G's geometric fabric allows us to control the robot in a way that imposes joint limit constraints, avoids self-collisions and environment collisions, and guarantees globally stable motion. It also exposes a low-dimensional action space and provides redundancy resolution.
@inproceedings{lum2024dextrahg,
title = {Dextr{AH}-G: Pixels-to-Action Dexterous Arm-Hand Grasping with Geometric Fabrics},
author = {Tyler Ga Wei Lum and Martin Matak and Viktor Makoviychuk and Ankur Handa and Arthur Allshire and Tucker Hermans and Nathan D. Ratliff and Karl Van Wyk},
booktitle = {8th Annual Conference on Robot Learning},
year = {2024},
url = {https://openreview.net/forum?id=S2Jwb0i7HN}
}