Active perception for robot manipulation

Robots can learn about objects by interacting with them: a touch reveals a hidden surface, a squeeze indicates softness, and a tap produces sounds that help identify a material. We develop methods that combine vision, touch, force sensing, and sound to estimate object shape and physical properties. A central question is how a robot should choose its next exploratory action to reduce uncertainty and gather information useful for manipulation.

Our work spans reconstructing object shapes from partial views and exploratory touches, recognizing deformable objects, and estimating material properties using standard robot grippers. We also investigate how gripper design and the way an action is performed influence what a robot can perceive. These approaches support more reliable grasping and handling of unfamiliar objects, with applications such as automated material sorting.

Related publications:

• Pliska, M., Patni, S. P., Mares, M., Stoudek, P., Straka, Z., Stepanova, K. and Hoffmann, M. (2024), 'Single-grasp deformable object discrimination: the effect of gripper morphology, sensing modalities, and action parameters', IEEE Transactions on Robotics  40, 4414 - 4426. [DOI - IEEE Xplore][pdf-arxiv][youtube-video][data: https://osf.io/zetg3/]

• Rustler, L.; Matas, J. & Hoffmann, M. (2023), Efficient Visuo-Haptic Object Shape Completion for Robot Manipulation, in 'Intelligent Robots and Systems (IROS), IEEE/RSJ International Conference on', pp. 3121-3128.  [pdf-arxiv][youtube-video]

• Rustler, L., Lundell, J., Behrens, J. K., Kyrki, V., & Hoffmann, M. (2022). 'Active Visuo-Haptic Object Shape Completion'. IEEE Robotics and Automation Letters 7 (2), 5254-5261. [DOI IEEE Xplore][arxiv][code and data][youtube-video]

• Kruzliak, A.; Hartvich, J.; Patni, S. P.; Rustler, L.; Behrens, J. K.; Abu-Dakka, F. J.; Mikolajczyk, K.; Kyrki, V. & Hoffmann, M. (2024). Interactive Learning of Physical Object Properties Through Robot Manipulation and Database of Object Measurements, in 'Intelligent Robots and Systems (IROS), IEEE/RSJ International Conference on', pp. 7596-7603. [DOI-IEEE Xplore][pdf-arxiv] [youtube-video][code][data - object database]

• Patni, S. P., Stoudek, P., Chlup, H. and Hoffmann, M. (2024), 'Online elasticity estimation and material sorting using standard robot grippers', The International Journal of Advanced Manufacturing Technology, 1--19. [DOI, Open Access] [video1 - elasticity estimation]  [video2 - online material sorting] [data: https://osf.io/gec6s/]

• Dimiccoli, M., Patni, S., Hoffmann, M. and Moreno-Noguer, F. (2022), Recognizing object surface material from impact sounds for robot manipulation, in 'Intelligent Robots and Systems (IROS), IEEE/RSJ International Conference on'.  [IEEE Xplore][youtube-video]