TieBot: Learning to Knot a Tie from Visual Demonstration through a Real-to-Sim-to-Real Approach
Abstract:
The tie-knotting task is highly challenging due to the tie's high deformation and long-horizon manipulation actions. This work presents TieBot, a Real-to-Sim-to-Real learning from visual demonstration system for the robots to learn to knot a tie. We introduce the Hierarchical Feature Matching approach to estimate a sequence of tie's meshes from the demonstration video. With these estimated meshes used as subgoals, we propose to learn feasible action sequences to achieve these subgoals from point clouds with a teacher-student training paradigm in the simulation. Lastly, our pipeline learns a residual policy when the learned policy is applied to real-world execution, mitigating the Sim2Real gap. We demonstrate the effectiveness of TieBot in simulation and the real world. In the real-world experiment, a dual-arm robot successfully knots a tie.
human demonstrations
![](https://www.google.com/images/icons/product/drive-32.png)
first tie-knotting
![](https://www.google.com/images/icons/product/drive-32.png)
second tie-knotting
![](https://www.google.com/images/icons/product/drive-32.png)
towel-folding
feature matching examples
first tie-knotting
second tie-knotting
towel-folding
keypoint detection examples
first tie-knotting
second tie-knotting
real2sim result examples
first tie-knotting
second tie-knotting
towel-folding
![](https://www.google.com/images/icons/product/drive-32.png)
real-world experiment
We test our real-world policy 10 times. Each time the initial positions of the tie are slightly perturbed about 5cm. The final success rate is 50%.