GraphGarment: Learning Garment Dynamics for Bimanual Cloth Manipulation Tasks
Accepted to 2025 IEEE/RSJ International Conference on Intelligent Robots and Systems
(Under Review for AIS)
Embedded Files
Introduction
Introduction
The deformable nature of fabrics makes tasks like garment hanging difficult for robots in household, healthcare, and industrial environments.
The paper introduces GraphGarment, a novel method that models garment dynamics based on robot control inputs to aid manipulation tasks.
It represents the interactions between the robot end-effector and the garment using graphs.
A graph neural network (GNN) is employed to learn a dynamics model that predicts the next state of the garment given the current state and action in simulation.
A residual model is proposed to compensate for prediction errors, thereby improving performance in real-world scenarios.
The learned dynamics model is used within a model-based action sampling strategy to manipulate the garment into a pre-hanging configuration for hanging tasks.
Challenges of Cloth Manipulation
Garments are highly deformable with various complex configurations, making dynamics learning challenging. Recent research uses data-driven methods to learn the dynamics model of the garment. However, the model learned in simulation is difficult to apply directly to the real-world due to the large sim-to-real gap(size, appearance, texture, and mechanical difference).
Garment Dynamics Learning
Data Acquisition
Unity Environment
Unity Environment
We use Unity with the Obi Cloth Physics engine, based for data collection, garment dynamics and bimanual garment-hanging with expert demonstrations captured via a VR system using an HTC VIVE controller.
Experiment Garment
Experiment Garment
This figure shows the experiment's target garments, including simulated (Garment 1, Garment 2, Garment 3) and real garment (Garment 1, Garment 2, Garment 3).
Graph Representation
Convert the garment point cloud into a graph, where each vertex includes a 3D position and a type
For grasp-related tasks, only edges among grasped areas and action nodes are constructed, allowing the prediction of the next grasped area.
With this design choice, the information is only updated at these points, enabling us to predict the grasped area in the next timestep without considering the unnecessary sections of the garment’s main body
Forward Graph Neural Network
•We use an Encoder-Processor-Decoder GNN with MLP-based encoders and message-passing processors to predict the next 3D positions of garment grasped areas from the current state and action.
•The model is trained in a supervised manner using mean squared error loss between predicted and ground truth positions.
Residual Model for Sim-to-real Transfer
Instead of fine-tuning the entire GNN model, we aim to apply an offset value to refine local predictions, effectively bridging the sim-to-real gap.
Specifically, we use the PointNet-based architecture for the training of the residual model. It processes the point cloud to extract local and global features, embeds the action, and predicts offsets to correct the point cloud.
Garment Pre-hanging Adjustment for Hanging
Garment Pre-hanging Adjustment for Hanging
Learning from Demonstration for Hanging
A model-based action sampling strategy for garment pre-hanging adjustment.
It evaluates candidate actions using a dynamics model to select the grasping action leading to an ideal pre-hanging configuration.
We apply Learning-from-Demonstration (LfD) to replicate expert behaviours
A Dynamic Movement Primitive (DMP) is then to replicate the expert hanging demonstration
Experiments Setup
Simulation
Evaluation of GraphGarment for garment next state prediction
Evaluation of GraphGarment for the Garment Hanging
Real-World
To obtain the garment point cloud, we use grounded sam with prompt 'garment' to segment the garment pixel in 2-D images. By aligning the RGB images and depth images, the garment pixel is then projected into 3-D point cloud with the camera intrinsics. A Baxter robot is applied for the manipulation of the Garment.
Garment Next State Prediction Experiment
Garment Next State Prediction Experiment
Garment Hanging Experiment
Detailed Presentation Video
Detailed Presentation Video
Page updated
Google Sites
Report abuse