FabricFlowNet: Bimanual Cloth Manipulation with a Flow-based Policy
Thomas Weng, Sujay Bajracharya, Yufei Wang, Khush Agrawal, David Held
Robotics Institute, Carnegie Mellon University, Pittsburgh, PA
[arXiv] [OpenReview] [Poster] [Code]
This work as has been accepted at CoRL 2021.
Abstract
We address the problem of goal-directed cloth manipulation, a challenging task due to the deformability of cloth. Our insight is that optical flow, a technique normally used for motion estimation in video, can also provide an effective representation for corresponding cloth poses across observation and goal images. We introduce FabricFlowNet (FFN), a cloth manipulation policy that leverages flow as both an input and as an action representation to improve performance. FabricFlowNet also elegantly switches between dual-arm and single-arm actions based on the desired goal. We show that FabricFlowNet significantly outperforms state-of-the-art model-free and model-based cloth manipulation policies. We also present real-world experiments on a bimanual system, demonstrating effective sim-to-real transfer. Finally, we show that our method generalizes when trained on a single square cloth to other cloth shapes, such as T-shirts and rectangular cloths.
Videos of FabricFlowNet Executing Square Cloth Goals
Our method is trained in simulation and transferred to the real world.
![](https://www.google.com/images/icons/product/drive-32.png)
Goal (All Corners In)
Achieved
![](https://www.google.com/images/icons/product/drive-32.png)
Goal (Opposite corners in)
Achieved
![](https://www.google.com/images/icons/product/drive-32.png)
Goal (Test 26)
Achieved
![](https://www.google.com/images/icons/product/drive-32.png)
Goal (Two side horizontal)
Achieved
![](https://www.google.com/images/icons/product/drive-32.png)
Goal (Triangle)
Achieved
![](https://www.google.com/images/icons/product/drive-32.png)
Goal (Test 6)
Achieved
![](https://www.google.com/images/icons/product/drive-32.png)
Goal (Test 7)
Achieved
![](https://www.google.com/images/icons/product/drive-32.png)
Goal (Test 8)
Achieved
![](https://www.google.com/images/icons/product/drive-32.png)
Goal (Test 9)
Achieved
![](https://www.google.com/images/icons/product/drive-32.png)
Goal (Test 10)
Achieved
![](https://www.google.com/images/icons/product/drive-32.png)
Goal (Test 27)
Achieved
![](https://www.google.com/images/icons/product/drive-32.png)
Goal (Test 11)
Achieved
![](https://www.google.com/images/icons/product/drive-32.png)
Goal (Test 24)
Achieved
![](https://www.google.com/images/icons/product/drive-32.png)
Goal (Test 25)
Achieved
![](https://www.google.com/images/icons/product/drive-32.png)
Goal (One corner in)
Achieved
![](https://www.google.com/images/icons/product/drive-32.png)
Goal (Double Rectangle)
Achieved
![](https://www.google.com/images/icons/product/drive-32.png)
Goal (Double Triangle)
Achieved
Videos of FabricFlowNet Generalizing to Rectangular Cloth and T-Shirt Goals
Our method is trained only on a square cloth in simulation and generalizes to other cloth shapes.
![](https://www.google.com/images/icons/product/drive-32.png)
Goal (T-shirt horizontal)
Achieved
![](https://www.google.com/images/icons/product/drive-32.png)
Goal (T-shirt vertical)
Achieved
![](https://www.google.com/images/icons/product/drive-32.png)
Goal (T-shirt full)
Achieved
![](https://www.google.com/images/icons/product/drive-32.png)
Goal (Rectangle two side horizontal)
Achieved
![](https://www.google.com/images/icons/product/drive-32.png)
Goal (Rectangle one corner)
Achieved
![](https://www.google.com/images/icons/product/drive-32.png)
Goal (Rectangle two side vertical)
Achieved
![](https://www.google.com/images/icons/product/drive-32.png)
Goal (Rectangle horizontal)
Achieved
![](https://www.google.com/images/icons/product/drive-32.png)
Goal (Rectangle vertical)
Achieved
Acknowledgements
This work was supported by the National Science Foundation (NSF) Smart and Autonomous Systems Program (IIS-1849154), a NSF CAREER Award (IIS-2046491), LG Electronics, and a NSF Graduate Research Fellowship (DGE-1745016).