Bimanual Manipulation of Deformable Bags Using Structure of Interest-based Latent Dynamics Model
Peng Zhou, Pai Zheng, Jiaming Qi, Chenxi Li, Chenguang Yang, David Navarro-Alarcon, and Jia Pan
The manipulation of deformable objects by robotic systems presents significant challenges due to their complex and infinite-dimensional configuration spaces. This paper introduces a novel approach to deformable object manipulation (DOM) by emphasizing the identification and manipulation of structures of interest (SOIs) in deformable fabric bags. We propose a bimanual manipulation framework that leverages a graph neural network (GNN)-based latent dynamics model to succinctly represent and predict the behavior of these SOIs. Our approach involves constructing a graph representation from partial point cloud data of the object and learning the latent dynamics model that effectively captures the essential deformations of the fabric bag within a reduced computational space. By integrating this latent dynamics model with model predictive control (MPC), we enable robotic manipulators to perform precise and stable manipulation tasks focused on the SOIs. We validate our new framework through various experiments that demonstrate its efficacy in manipulating deformable bags. Our contributions not only address the complexities inherent in DOM but also provide new perspectives and methodologies for enhancing robotic interactions with deformable materials by concentrating on their critical structural elements.
Note: In all of our experiments, the final goal states for the Structures of Interest (SOI) are indicated by green particles. These states are randomly sampled from the training dataset's particles and then projected onto the camera plane from the current perspective, utilizing the appropriate extrinsic and intrinsic parameters. Concurrently, the blue particles—which are also projected onto the camera plane using the same parameters—are the predicted states from the SOI latent dynamics system, with the action sequences being optimized through Model Predictive Control (MPC).
The previously uploaded videos are available via https://sites.google.com/view/bagbot-v0
Python code snippet of how our method achieves the DOM tasks
