FBI: Learning Dexterous In-hand Manipulation with Dynamic Visuotactile Shortcut Policy

We propose Flow Before Imitation (FBI), an imitation learning framework integrating tactile and visual information through motion dynamics. 

Dexterous in-hand manipulation is a long-standing challenge in robotics due to complex contact dynamics and partial observability. While humans synergize vision and touch for such tasks, robotic approaches often prioritize one modality, therefore limiting adaptability. This paper introduces Flow Before Imitation (FBI), a visuotactile imitation learning framework that dynamically fuses tactile interactions with visual observations through motion dynamics. Unlike prior static fusion methods, FBI establishes a causal link between tactile signals and object motion via a dynamics-aware latent model. FBI employs a transformer-based interaction module to fuse flow-derived tactile features with visual inputs, training a one-step diffusion policy for real-time execution. Extensive experiments demonstrate that the proposed method outperforms the baseline methods in both simulation and the real world on two customized in-hand manipulation tasks and three standard dexterous manipulation tasks.

 FBI enables two operational modes with or without physical tactile sensors in the real-world experiments, largely extending the application scenarios.

In simulated tasks, FBI achieves 16.6% (Vision-only)  to 18.4% (Visuotactile) improvement compared to DP3, with a smaller variance. In the In-hand Reorientation task, FBI surpasses DP3 with a 19.3% (Vision-only) to 21.4% (Visuotactile) improvement averaged over 9 different objects, indicating that after fusing dense contact states with visual information, our algorithm demonstrates superior performance, particularly in handling more complex dexterous tasks.

In real-world tasks, FBI outperforms all baselines in all tasks, achieving 15.0% (Vision-only) to 16.5% (Visuotactile) improvement compared to the previous SOTA baseline, ManiCM.