Embodied Uncertainty-Aware Object Segmentation

Xiaolin Fang, Leslie Pack Kaelbling, Tomás Lozano-Pérez

MIT CSAIL

{xiaolinf,lpk,tlp}@csail.mit.edu

Abstract

We introduce uncertainty-aware object instance segmentation (UncOS) and demonstrate its usefulness for embodied interactive segmentation. To deal with uncertainty in robot perception, we propose a method for generating a hypothesis distribution of object segmentation. We obtain a set of region-factored segmentation hypotheses together with confidence estimates by making multiple queries of large pre-trained models. This process can produce segmentation results that achieve state-of-the-art performance on unseen object segmentation problems. The output can also serve as input to a belief-driven process for selecting robot actions to perturb the scene to reduce ambiguity. We demonstrate the effectiveness of this method in real-robot experiments.

EOS

A method for converting a segmentation distribution into a distribution over world models

and using that for selecting robot perturbation actions to disambiguate the segmentation

UncOS

An active prompting strategy for combining promptable top-down and bottom-up

pre-trained object instance segmentation methods to obtain a distribution over image-segmentation hypotheses

Qualitative Results on OCID Dataset

UncOS takes an RGB (+Depth optionally) image and generates multiple segmentation hypotheses of the scene and their associated confidence.

To use it as an unseen object instance segmentation (UOIS) method, we take the most likely/confident hypothesis reported by UncOS.

The results shown below are the most confident hypothesis reported by UncOS on the OCID testset. These are NOT carefully selected (uniformly sampled from the test set).

Click on the images to expand and see more results.

Embodied Segmentation based on UncOS Result

We use a robot to selectively poke objects in the scene using a simple greedy strategy that attempts to select a small perturbation that will maximize information gain.

Regions correctly segmented after embodied interaction are highlighted with green circles (errors highlighted with red circles).

BibTeX

@inproceedings{Fang2024Uncos, title={{Embodied Uncertainty-Aware Object Segmentation}}, author={Xiaolin Fang and Leslie Pack Kaelbling and Tomás Lozano-Pérez}, booktitle={IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)}, year={2024},}

Page updated

Google Sites

Report abuse