Fail2Progress: Learning from Failures with Stein Variational Inference for Robot Manipulation
Fail2Progress: Learning from Failures with Stein Variational Inference for Robot Manipulation
Yixuan Huang 1, Novella Alvina 1, Mohanraj Devendran Shanthi 1 , Tucker Hermans 1|2
1 University of Utah, 2 NVIDIA Research
Skill effect models for long-horizon manipulation tasks are prone to failures in conditions not covered by training data distributions. Therefore, enabling robots to reason about and learn from failures is necessary. We investigate the problem of efficiently generating a dataset targeted to observed failures. After fine-tuning a skill effect model on this dataset, we evaluate the extent to which the model can recover from failures and minimize future failures. We propose Fail2Progress, an approach that leverages Stein variational inference to generate multiple simulation environments in parallel, enabling efficient data sample generation similar to observed failures. Our method is capable of handling several challenging mobile manipulation tasks, including transporting multiple objects, organizing a constrained shelf, and tabletop organization. Through large-scale simulation and real-world experiments, we demonstrate that our approach excels at learning from failures across different numbers of objects. Furthermore, we show that Fail2Progress outperforms several baselines.
Related research
[1] Points2Plans | Huang, Y., Agia, C., Wu, J., Hermans, T., & Bohg, J (2025). IEEE International Conference on Robotics and Automation (ICRA).
[2] STAP: Sequencing Task-Agnostic Policies | Agia, C., Migimatsu, T., Wu, J., & Bohg, J. (2023). IEEE International Conference on Robotics and Automation (ICRA) (pp. 7951-7958).
[3] Search-Based Task Planning with Learned Skill Effect Models for Lifelong Robotic Manipulation | Liang, J., Sharma, M., Lagrassa, A., Vats, S., Saxena, S., & Kroemer, O. (2022). IEEE International Conference on Robotics and Automation (ICRA).