Fail2Progress: Learning from Failures with Stein Variational Inference for Robot Manipulation
Fail2Progress: Learning from Failures with Stein Variational Inference for Robot Manipulation
Yixuan Huang 1, Novella Alvina 1, Mohanraj Devendran Shanthi 1 , Tucker Hermans 1|2
1 University of Utah, 2 NVIDIA Research
Skill sequencing models for long-horizon manipulation tasks are prone to failures in conditions not covered by training data distributions. Therefore, enabling robots to reason about and learn from failures is necessary. We investigate the problem of efficiently generating a dataset based on observed failures. After fine-tuning a skill sequencing model on this dataset, we evaluate the extent to which the model can recover from failures and minimize future failures. We propose Fail2Progress, an approach that leverages Stein variational inference to generate multiple simulation environments in parallel, enabling efficient data sample generation similar to observed failures. Our method is capable of handling several challenging mobile manipulation tasks including transporting multiple objects, organizing a constrained shelf, and retrieving occluded objects from a drawer. Through large-scale simulation and real-world experiments, we demonstrate that our approach excels at learning from failures across different numbers of objects. Furthermore, we show that Fail2Progress outperforms several baselines in terms of performance and efficiency.
Task 1: Multi-object Transport
Description: The robot is tasked with packing groceries and placing them on the table. It places all four groceries inside a bag. In the failure case, the robot places the bag on the ground instead of the table, failing the task. After fine-tuning the model with a targeted dataset, Fail2Progress moves the bag to the table.
Task 2: Drawer Retrieval
Description: The robot first opens the drawers, places a snack in one of the drawers, and then closes them. The next day, the robot is tasked with retrieving an occluded snack based on its memory. In the failure case, the robot opens the wrong drawer due to inaccurate memory of the snack’s location. After fine-tuning the model on a targeted dataset, Fail2Progress succeeds by recalling the correct location of the snack, opening the appropriate drawer, and grasping the snack box.
Task 3: Constrained Packing
Description: The robot is tasked with organizing a shelf by placing a stack of cups on a constrained shelf. In the failure case, the robot fails to clear enough space, causing a collision between the cups and the wipes. After learning from the failure, Fail2Progress first pushes the wipes aside to create sufficient space for the cups, then places them on the shelf.
Task 4: Hierarchical Tabletop Organization
Description: The robot is tasked with organizing the cups and capsules. It places several capsules into their corresponding cups. In the failure case, the robot fails to recognize the correlation between cups and capsules, resulting in collisions. After learning from this failure, Fail2Progress successfully completes this task by understanding that the capsules will move with their corresponding cups.
We demonstrate how our approach, Fail2Progress, generalizes to different numbers and shapes of objects, as well as different tables. Specifically, the model is fine-tuned only on failure cases with 3 objects but is able to generalize to scenarios involving 3-6 diverse objects on two tables.
Related research
[1] Points2Plans | Huang, Y., Agia, C., Wu, J., Hermans, T., & Bohg, J (2025). IEEE International Conference on Robotics and Automation (ICRA).
[2] STAP: Sequencing Task-Agnostic Policies | Agia, C., Migimatsu, T., Wu, J., & Bohg, J. (2023). IEEE International Conference on Robotics and Automation (ICRA) (pp. 7951-7958).
[3] Search-Based Task Planning with Learned Skill Effect Models for Lifelong Robotic Manipulation | Liang, J., Sharma, M., Lagrassa, A., Vats, S., Saxena, S., & Kroemer, O. (2022). IEEE International Conference on Robotics and Automation (ICRA).