Reasoning with Scene Graphs for Robot Planning under Partial Observability

Abstract

Robot planning in partially observable domains is difficult, because a robot needs to estimate the current state and plan actions at the same time. When the domain includes many objects, reasoning about the objects and their relationships makes robot planning even more difficult. In this paper, we develop an algorithm called scene analysis for robot planning (SARP) that enables robots to reason with visual contextual information toward achieving long-term goals under uncertainty. SARP constructs scene graphs, a factored representation of objects and their relations, using images captured from different positions, and reasons with them to enable context-aware robot planning under partial observability. Experiments have been conducted using multiple 3D environments in simulation, and a dataset collected by a real robot. In comparison to standard robot planning and scene analysis methods, in a target search domain, SARP improves both efficiency and accuracy in task completion.

SARP Overview

An overview of SARP, where the robot takes an action using the policy, and receives an observation from the world, updating the belief using the action and observation. In each timestep, a graph is accumulated using the perception sensor to bias the belief. This biased belief provides the contextual information to the policy. Top dashed lines show that the belief biasing not occurring in every iteration. The bottom dashed line indicates that the scene graph networked is trained offline.

Presentation

Slides

Supplementary Video