【Paper】

Abstract

Bimanual dexterous manipulation is a critical yet underexplored area in robotics. Its high-dimensional action space and inherent task complexity present significant challenges for policy learning, and the limited task diversity in existing benchmarks hinders general-purpose skill development. Existing approaches largely depend on reinforcement learning, often constrained by intricately designed reward functions tailored to a narrow set of tasks. In this work, we present a novel approach for efficiently learning diverse bimanual dexterous skills from abundant human demonstrations. Specifically, we introduce BiDexHD (Bimanual Dexterity from Human Demonstrations), a framework that unifies task construction from existing bimanual datasets and employs teacher-student policy learning to address all tasks. The teacher learns state-based policies using a general two-stage reward function across tasks with shared behaviors, while the student distills the learned multi-task policies into a vision-based policy. With BiDexHD, scalable learning of numerous bimanual dexterous skills from auto-constructed tasks becomes feasible, offering promising advances toward universal bimanual dexterous manipulation. Our empirical evaluation on the TACO dataset, spanning 141 tasks across six categories, demonstrates a task fulfillment rate of 74.59% on trained tasks and 51.07% on unseen tasks, showcasing the effectiveness and competitive zero-shot generalization capabilities of BiDexHD.

BiDexHD: Unified and Scalable Bimanual Framework

The three-phase framework, BiDexHD, unifies constructing and solving tasks from human bimanual datasets instead of existing benchmarks, providing a robust and scalable solution to diverse challenging bimanual manipulation tasks.

• In phase one, BiDexHD constructs each bimanual task from a human demonstration. 

• In phase two, BiDexHD learns diverse state-based policies from a generally designed two-stage reward function via multi-task reinforcement learning. 

• In phase three, A group of learned policies are then distilled into a vision-based policy for inference.

In particular, general two-stage multi-task reinforcement learning is adopted for acquiring expert state-based bimanual skills.

• At stage zero, for each task, all joint poses are initialized at zero pose and a pair of tool objects are initialized at a fixed pose.

• At stage one, the approaching reward encourages both hands to get close to their grasping centers, and the lifting reward along with the bonus incentivizes moving both objects to their reference poses respectively.

• After simulation alignment, at stage two dual hands will manipulate objects under the guidance of the tracking reward.

Diverse Bimanual Skills of TACO tasks

Quantitative Results of TACO tasks

Diverse Bimanual Skills of Arctic tasks