Fail2Progress: Learning from Real-World Robot Failures with Stein Variational Inference
Fail2Progress: Learning from Real-World Robot Failures with Stein Variational Inference
Yixuan Huang 1, Novella Alvina 1, Mohanraj Devendran Shanthi 1 , Tucker Hermans 1|2
1 University of Utah, 2 NVIDIA Research
Conference on Robot Learning (CoRL) 2025
Skill effect models for long-horizon manipulation tasks are prone to failures in conditions not covered by training data distributions. Therefore, enabling robots to reason about and learn from failures is necessary. We investigate the problem of efficiently generating a dataset targeted to observed failures. After fine-tuning a skill effect model on this dataset, we evaluate the extent to which the model can recover from failures and minimize future failures. We propose Fail2Progress, an approach that leverages Stein variational inference to generate multiple simulation environments in parallel, enabling efficient data sample generation similar to observed failures. Our method is capable of handling several challenging mobile manipulation tasks, including transporting multiple objects, organizing a constrained shelf, and tabletop organization. Through large-scale simulation and real-world experiments, we demonstrate that our approach excels at learning from failures across different numbers of objects. Furthermore, we show that Fail2Progress outperforms several baselines.
Citation
If you found this work interesting, please consider citing:
@inproceedings{Huang-2025-fail2progress,
title = {Fail2Progress: Learning from Real-World Robot Failures with Stein Variational Inference},
author = {Huang, Yixuan and Alvina, Novella and Devendran Shanthi, Mohanraj and Hermans, Tucker},
booktitle = {Conference on Robot Learning (CoRL), 2025},
year = {2025}
}
Related research
[1] Points2Plans | Huang, Y., Agia, C., Wu, J., Hermans, T., & Bohg, J (2025). IEEE International Conference on Robotics and Automation (ICRA).
[2] STAP: Sequencing Task-Agnostic Policies | Agia, C., Migimatsu, T., Wu, J., & Bohg, J. (2023). IEEE International Conference on Robotics and Automation (ICRA) (pp. 7951-7958).
[3] Search-Based Task Planning with Learned Skill Effect Models for Lifelong Robotic Manipulation | Liang, J., Sharma, M., Lagrassa, A., Vats, S., Saxena, S., & Kroemer, O. (2022). IEEE International Conference on Robotics and Automation (ICRA).
Acknowledgements