Learning robot manipulation from abundant human videos offers a scalable alternative to costly robot-specific data collection. However, domain gaps across visual, morphological, and physical aspects hinder direct imitation. To effectively bridge the domain gap, we propose ImMimic, an embodiment-agnostic co-training framework that leverages both human videos and a small amount of teleoperated robot demonstrations.  ImMimic uses Dynamic Time Warping (DTW) with either action- or visual-based mapping to map retargeted human hand poses to robot joints, followed by MixUp interpolation between paired human and robot trajectories. Our key insights are (1) retargeted human hand trajectories provide informative action labels, and (2) interpolation over the mapped data creates intermediate domains that facilitate smooth domain adaptation during co-training. Evaluations on four real-world manipulation tasks (Pick and Place, Push, Hammer, Flip) across four robotic embodiments (Robotiq, Fin Ray, Allegro, Ability) show that ImMimic improves task success rates and execution smoothness, highlighting its efficacy to bridge the domain gap for robust robot manipulation.

Our key insights are:

1. Beyond the visual contexts, the retargeted human hand trajectories can serve as action labels for human demonstrations.

2. Creating intermediate domains via interpolation leads to robust adaptation.

3. Establishing an effective mapping between human and robot data for interpolation is essential for co-training. 

Method

We collect, map, and interpolate human and robot data through the following steps:

• (a) Robot demonstrations are collected using visual teleoperation.

• (b) Human actions are extracted and retargeted from videos.

• (c, d) Using visual or action-based Dynamic Time Warping (DTW), we map the retargeted human and robot trajectories.

• (e) Through MixUp, the mapped human-robot pairs are interpolated in both the latent space and the action space, to generate new interpolated human data for training.

Finally, the interpolated human data is co-trained alongside the robot data. See the diagram below for an overview of the co-training pipeline.

1. For robot demonstration, we train the policy using agent-view and wrist-view images encoded by ResNet along with proprioception. All are combined into the observation condition to predict future actions. 

2.  For human demonstration, we train the same diffusion policy using human videos.

   • A hand pose retargeting module generates retargeted action which serve as both the future action and proprioception for training. 

   • Mapping with DTW, we apply MixUp for human data with paired robot data.

   • The interpolation enables human data to smoothly adapt to the robot data.

3. The model is optimized upon the sum of reconstruction losses.

We validate ImMimic on four diverse manipulation tasks (Pick and Place, Push, Hammer, Flip) across four robotic embodiments (Robotiq Gripper, Fin Ray Gripper, Allegro Hand, Ability Hand). Below are the results for ImMimic-A (action-based mapping):

Blue indicates success, and Red indicates failure.

ImMimic significantly and consistently improves the task performance over baseline methods (Co-Training, Random Mapping, Robot-Only). Below are the results:

Blue indicates success, and Red indicates failure.

We found out that interpolation with action-based mapping (ImMimic-A) yields better performance than interpolation with vision-based mapping (ImMimic-V) in our setting. Below are the results:

Blue indicates success, and Red indicates failure.

We also show that scale and diversity of human demonstrations enhance learning performance. Below are the results: