Cross-Embodiment Dexterous Grasping with Reinforcement Learning

CrossDex employs a unified observation and action space to facilitate the learning of a universal policy across various dexterous hands. Rather than relying on joint angles specific to each hand, our policy utilizes the positions of the fingertips and palm to discern the spatial relationship between the hand and the object. Actions are represented using eigengrasps from the MANO hand model, which are mapped to position targets of each hand's PD controller through a retargeting process. This design, akin to teleoperation, enables consistent control across different dexterous hands. The policy is trained using reinforcement learning within a cross-embodiment simulation environment built on IsaacGym. To learn a vision-based policy, we substitute the object pose in this pipeline with the object's point cloud.