FabricFlowNet: Bimanual Cloth Manipulation with a Flow-based Policy

Thomas Weng, Sujay Bajracharya, Yufei Wang, Khush Agrawal, David Held

Robotics Institute, Carnegie Mellon University, Pittsburgh, PA

This work as has been accepted at CoRL 2021.

Abstract

We address the problem of goal-directed cloth manipulation, a challenging task due to the deformability of cloth. Our insight is that optical flow, a technique normally used for motion estimation in video, can also provide an effective representation for corresponding cloth poses across observation and goal images. We introduce FabricFlowNet (FFN), a cloth manipulation policy that leverages flow as both an input and as an action representation to improve performance. FabricFlowNet also elegantly switches between dual-arm and single-arm actions based on the desired goal. We show that FabricFlowNet significantly outperforms state-of-the-art model-free and model-based cloth manipulation policies. We also present real-world experiments on a bimanual system, demonstrating effective sim-to-real transfer. Finally, we show that our method generalizes when trained on a single square cloth to other cloth shapes, such as T-shirts and rectangular cloths.

Videos of FabricFlowNet Executing Square Cloth Goals

Our method is trained in simulation and transferred to the real world.

ms_all_corn_in_merged.mp4

Goal (All Corners In)

Achieved

ms_opp_corn_in_merged.mp4

Goal (Opposite corners in)

Achieved

Copy of towel_train_26_high_gray_dual.mp4

Goal (Test 26)

Achieved

ms_two_side_horz_merged_2.mp4

Goal (Two side horizontal)

Achieved

Copy of triangle_0_high_gray_single.mp4

Goal (Triangle)

Achieved

Copy of towel_train_6_high_gray_single.mp4

Goal (Test 6)

Achieved

Copy of towel_train_7_high_gray_single.mp4

Goal (Test 7)

Achieved

Copy of towel_train_8_high_gray_single.mp4

Goal (Test 8)

Achieved

Copy of towel_train_9_high_gray_single.mp4

Goal (Test 9)

Achieved

Copy of towel_train_10_high_gray_single.mp4

Goal (Test 10)

Achieved

Copy of towel_train_27_high_gray_dual.mp4

Goal (Test 27)

Achieved

Copy of towel_train_11_high_gray_single.mp4

Goal (Test 11)

Achieved

Copy of towel_train_24_high_gray_dual.mp4

Goal (Test 24)

Achieved

Copy of towel_train_25_high_gray_dual.mp4

Goal (Test 25)

Achieved

Copy of one_corn_in_0_high_gray_single.mp4

Goal (One corner in)

Achieved

ms_double_rect_merged.mp4

Goal (Double Rectangle)

Achieved

ms_double_tri_merged.mp4

Goal (Double Triangle)

Achieved

Videos of FabricFlowNet Generalizing to Rectangular Cloth and T-Shirt Goals

Our method is trained only on a square cloth in simulation and generalizes to other cloth shapes.

Copy of horz_dual.mp4

Goal (T-shirt horizontal)

Achieved

Copy of tsf_dual.mp4

Goal (T-shirt vertical)

Achieved

tsh_full_merged.mp4

Goal (T-shirt full)

Achieved

rect_two_side_horz_merged.mp4

Goal (Rectangle two side horizontal)

Achieved

Copy of corner_single.mp4

Goal (Rectangle one corner)

Achieved

rect_two_side_vert_merged.mp4

Goal (Rectangle two side vertical)

Achieved

Copy of horz_dual.mp4

Goal (Rectangle horizontal)

Achieved

Copy of vert_dual.mp4

Goal (Rectangle vertical)

Achieved

Acknowledgements

This work was supported by the National Science Foundation (NSF) Smart and Autonomous Systems Program (IIS-1849154), a NSF CAREER Award (IIS-2046491), LG Electronics, and a NSF Graduate Research Fellowship (DGE-1745016).