Iterative Interactive Modeling for Knotting Plastic Bags

Chongkai Gao, Zekun Li, Haichuan Gao, Feng Chen

Department of Automation, Tsinghua University

Conference on Robot Learning (CoRL), 2022

paper, zhihu (In Chinese)

Video Introduction (with Audio)

Abstract

Deformable object manipulation has great research significance for the robotic community and numerous applications in daily life. In this work, we study how to knot plastic bags that are randomly dropped from the air with a dual arm robot and image input. The complex initial configuration and intricate physical properties of plastic bags pose challenges for reliable perception and planning. Directly knotting it from random initial states is difficult. In this work, we propose Iterative Interactive Modeling (IIM) to first adjust the plastic bag to a standing pose with imitation learning to establish a high-confidence keypoint skeleton model, then perform a set of learned motion primitives to knot it. We leverage spatial action maps to accomplish the iterative pick-and-place action and a graph convolutional network to evaluate the adjusted pose during the IIM process. In experiments, we achieve an 85.0% success rate in knotting 4 different plastic bags including one that has no demonstration.

The IIM process

Method

The robot is trained to iteratively adjust the plastic bag to a standing pose to build a complete and highconfidence keypoint model with the help of a task progress module, then tie the knot with a set of learned motion primitives. We use keypoint skeleton as the visual representation for plastic bags. 

At each step, the right camera gets a top-down view image, and the left camera gets a sequence of side view images. We use a spatial action maps module φ to get the grasping points, and use a sparse reconstructed point cloud to get the grasping depth. At the same time, we detect 2D keypoints from left images with the trained keypoint detecor to facilitate the task progress module G (a graph neural network) to determine which picking action to use, and when the adjusting pose is good enough to support building a 3D keypoint skeleton to tie the knot.

Results

 Success rates of different methods. Please refer to the paper and the above video for more results.