Mesh-based Dynamics with Occlusion Reasoning for Cloth Manipulation
RSS 2022
Zixuan Huang, Xingyu Lin, David Held
Carnegie Mellon University
Embedded Files
Abstract
Abstract
Self-occlusion is challenging for cloth manipulation as it makes it difficult to estimate the full state of the clothes. The robot needs to understand the whole structure of the cloth to fold or unfold the cloth into a goal configurations efficiently. We leverage the recent advances in pose estimation for clothes to build a system that reasons about occlusions explicitly. Specifically, we first learn a model to reconstruct the mesh of the cloth in both canonical and observation space, which we can use a prior on the occluded regions of the cloth. We then perform test-time fine-tuning using self-supervised losses to improve the reconstructed mesh further. The obtained full mesh allows us to learn a mesh-based dynamics model for planning. We also introduce a generalized version of cloth flattening for clothes with complex structures like a t-shirt with a front opening, which we refer to as cloth canonicalization. Our experiments show that our method significantly outperforms prior methods that don't account for occlusion explicitly.
Overview of Algorithm
Overview of Algorithm
Our proposed method can be seen as three stages:
Reconstruct a complete mesh of the clothes from a depth image.
Fine-tune the reconstructed mesh with two self-supervised losses.
Plan to achieve desired goal with a learned mesh-based dynamics model and the reconstructed mesh.
Visualizations of Reconstructed Meshes
Visualizations of Reconstructed Meshes
Our model reconstructs the full mesh of clothes given a single-view depth image. At test-time, we use two self-supervised losses to fine-tune the predicted mesh on-the-fly, which is shown to benefit the performance of downstream manipulation tasks (you can see it at the end of the website).
In the following section, we show the reconstructed meshes of crumpled clothes from 5 different categories. For each example, we demonstrate the quality of reconstructed mesh by interactive plot, as well as the fine-tuning process. As you can see, after fine-tuning, the predicted mesh becomes more aligned with the observation and also better capture the local geometric information.
Note:
These plots are interactive and might takes 1-2 minutes to load. If some plots cannot be displayed, please try reloading the page or click on the name of each clothes to view it separately.
For the 3D plots, you can drag them around to view the reconstructed mesh from different angles. Particularly, you can see how the occluded parts are reconstructed from below.
RGB is only for visualization. Our model takes depth as input.
Top
Front
Side
Fine-tuning Process
Gt mesh Fine-tuning predicted mesh
Top
Front
Side
Fine-tuning Process
Gt mesh Fine-tuning predicted mesh
Top
Front
Side
Fine-tuning Process
Gt mesh Fine-tuning predicted mesh
Top
Front
Side
Fine-tuning Process
Gt mesh Fine-tuning predicted mesh
Top
Front
Side
Fine-tuning Process
Gt mesh Fine-tuning predicted mesh
Real-world Experiments (smoothing)
Real-world Experiments (smoothing)
We deploy our proposed method on a Franka robot to conduct clothes smoothing. In the videos below, we show how we utilize the reconstructed mesh for planning and select a good action to execute. The actions are shown in the right plot as white arrows with red bodies. We use downsampled mesh during the rollout to accelerate the planning process.
1_output.mp42_output.mp40_output.mp4Task performance on Flattening and Canonicalization
Task performance on Flattening and Canonicalization
Our method is able to deal with different types of clothes and achieves the SOTA performance on flattening and canonicalization tasks. The objective of flattening task is to maximize the coverage of the clothes, while the goal of canonicalization is to deform the clothes in a way that it will match the canonical pose.
Page updated
Google Sites
Report abuse