Wenqi Ge1, Chao Tang1, Hong Zhang1,*

[1] Shenzhen Key Laboratory of Robotics and Computer Vision, SUSTech, Shenzhen, China.

[*] corresponding author: Hong Zhang.

Abstract: Household robots are often tasked with searching frequently moved objects for sub-tasks, making object search a challenging task. For household robots with access to pre-built maps identifying stationery items, accurately locating a general area for movable target objects can greatly enhance search efficiency. In similar contexts, previous methods depended on statistical correlations or used graph neural networks (GNN) to learn from images, aiming to roughly locate the target for search guidance. However, those methods are respectively constrained by environmental specificity and incomplete spatial information.  To address these limitations,  we propose \textbf{CSG-OS}. This novel commonsense scene graph-based object search framework is enhanced with commonsense knowledge via a large language model (LLM), which aligns target localization with human-like reasoning to inspire object search better. We train CSG-base target localization (CSG-TL), a core component of CSG-OS, on ScanNet dataset and evaluate for zero-shot performance cross on both ScanNet and AI2THOR simulator. Besides, the CSG-OS framework also achieves the SOTA performance on the AI2Thor simulator. We also deploy CSG-OS on the Jackal robot to validate its efficacy.

@misc{ge2024commonsense,

      title={Commonsense Scene Graph-based Target Localization for Object Search}, 

      author={Wenqi Ge and Chao Tang and Hong Zhang},

      year={2024},

      eprint={2404.00343},

      archivePrefix={arXiv},

      primaryClass={cs.RO}

}