Leveraging 3D Reconstruction for Mechanical Search on Cluttered Shelves

[ Information ].

International Conference on Robot Learning, 2023.

[ Authors ]

Seungyeon Kim, Young Hun Kim, Yonghyeon Lee, and Frank C. Park

[ Abstract ]

Finding and grasping a target object on a cluttered shelf is a challenging manipulation skill. When the target is occluded by other unknown objects and initially not visible, the robot needs to reposition the obstructing objects by pushing or pick-and-place actions. Additionally, a narrow shelf space filled with multiple objects leads to a very small collision-free space, limiting the manipulator's workspace. Existing works for the most part focus on developing specialized tools with suction grippers to overcome these challenges. In this paper, we use a standard two-finger gripper and 6-DoF grasping for pick-and-place and develop a mechanical search algorithm based on 3D reconstruction of the scene. Specifically, we define two functions, an existence function and a graspability function, and formulate a model-based optimal control problem. To estimate the two functions and construct approximate dynamics models for them, we exploit a pre-trained pushing dynamics model, depth rendering, and collision detection, which are all based on the 3D reconstruction results. Our methods not only show comparable performance to when objects are assumed to be known in simulation but also, through real-world experiments, show the real-world applicability and robustness of our method.