GraphGarment: Learning Garment Dynamics for Bimanual Cloth Manipulation Tasks

Accepted to 2025 IEEE/RSJ International Conference on Intelligent Robots and Systems

Introduction

The deformable nature of fabrics makes tasks like garment hanging difficult for robots in household, healthcare, and industrial environments.
The paper introduces GraphGarment, a novel method that models garment dynamics based on robot control inputs to aid manipulation tasks.
It represents the interactions between the robot end-effector and the garment using graphs.
A graph neural network (GNN) is employed to learn a dynamics model that predicts the next state of the garment given the current state and action in simulation.
A residual model is proposed to compensate for prediction errors, thereby improving performance in real-world scenarios.
The learned dynamics model is used within a model-based action sampling strategy to manipulate the garment into a pre-hanging configuration for hanging tasks.

Challenges of Cloth Manipulation

Garments are highly deformable with various complex configurations, making dynamics learning challenging. Recent research uses data-driven methods to learn the dynamics model of the garment. However, the model learned in simulation is difficult to apply directly to the real-world due to the large sim-to-real gap(size, appearance, texture, and mechanical difference).

Garment Dynamics Learning

Data Acquisition

Unity Environment

We use Unity with the Obi Cloth Physics engine, based for data collection, garment dynamics and bimanual garment-hanging with expert demonstrations captured via a VR system using an HTC VIVE controller.

Experiment Garment

This figure shows the experiment's target garments, including simulated (Garment 1, Garment 2, Garment 3) and real garment (Garment 1, Garment 2, Garment 3).

Graph Representation

Convert the garment point cloud into a graph, where each vertex includes a 3D position and a type
For grasp-related tasks, only edges among grasped areas and action nodes are constructed, allowing the prediction of the next grasped area.
With this design choice, the information is only updated at these points, enabling us to predict the grasped area in the next timestep without considering the unnecessary sections of the garment’s main body

Forward Graph Neural Network

•We use an Encoder-Processor-Decoder GNN with MLP-based encoders and message-passing processors to predict the next 3D positions of garment grasped areas from the current state and action.

•The model is trained in a supervised manner using mean squared error loss between predicted and ground truth positions.

Residual Model for Sim-to-real Transfer

Instead of fine-tuning the entire GNN model, we aim to apply an offset value to refine local predictions, effectively bridging the sim-to-real gap.

Specifically, we use the PointNet-based architecture for the training of the residual model. It processes the point cloud to extract local and global features, embeds the action, and predicts offsets to correct the point cloud.

Garment Pre-hanging Adjustment for Hanging

Learning from Demonstration for Hanging

A model-based action sampling strategy for garment pre-hanging adjustment.

It evaluates candidate actions using a dynamics model to select the grasping action leading to an ideal pre-hanging configuration.

We apply Learning-from-Demonstration (LfD) to replicate expert behaviours

A Dynamic Movement Primitive (DMP) is then to replicate the expert hanging demonstration

Experiments Setup

Simulation

Evaluation of GraphGarment for garment next state prediction

Evaluation of GraphGarment for the Garment Hanging

Real-World

To obtain the garment point cloud, we use grounded sam with prompt 'garment' to segment the garment pixel in 2-D images. By aligning the RGB images and depth images, the garment pixel is then projected into 3-D point cloud with the camera intrinsics. A Baxter robot is applied for the manipulation of the Garment.

Garment Next State Prediction Experiment

Garment Hanging Experiment

Detailed Presentation Video

Page updated

Google Sites

Report abuse