In-Hand Manipulation of Articulated Tools with Dexterous Robot Hands with Sim-to-Real Transfer

A privileged oracle is trained in simulation and distilled into a proprioceptive student policy for hardware deployment. Real-world tactile and motor torque signals are then used to train the (Cross CATFA module, which refines the transferred policy through cross-attention for contact-aware adaptation.

Technical overview of our pipeline for articulated in-hand manipulation. (a) Oracle training with privileged observations in simulation, followed by distillation into a proprioceptive base policy and real-world data collection with additional tactile and motor torque signals. (b) CATFA fuses these sensory modalities with policy intent via cross-attention, enabling online adaptation on hardware. This design establishes a disturbance-aware sim-to-real framework that refines contact interactions and improves robustness under articulation-dependent dynamics.