Learning Generalizable Language-Conditioned Cloth Manipulation from Long Demonstrations

Our framework consists of three stages. 

Stage 1: Skill Discovery from Demonstrations

In the first stage, we perform skill discovery from extensive cloth manipulation demonstrations by inputting language instructions and associated prompts into a large language model (LLM). Coupled with corresponding depth images, these inputs form a language-conditioned basic skill dataset, which is subsequently used for training the foundational skills.

Stage 2: Skill Learning with Transformer-Based Model

In the second stage,  using this dataset, a transformer-based model learns the skills. The model takes as input the language instruction and the current observation's depth image, and outputs a heatmap indicating the pick or place positions.

Stage 3: Task Planning and Composition

In the final stage, an LLM-based task planner composes the learned basic skills to address unseen multi-step manipulation tasks.

Our framework leverages LLM to extract fundamental skills from extensive demonstrations. The system receives a language instruction along with corresponding action steps and textual prompts as input. The LLM then automatically decomposes the task into a sequence of "pick" and "place" instructions. Each of these instructions is paired with an observation image, creating a language-conditioned basic skill dataset that supports subsequent skill learning.

By utilizing the diverse range of tasks found within long demonstrations, our approach can acquire a variety of fabric manipulation skills. These skills encompass the majority of "pick" and "place" actions across different primitive parts of the fabric. This comprehensive set of skills forms a robust foundation for generalizable manipulation tasks, enabling the system to handle a wide array of fabric handling scenarios effectively.