Knowledge-Graph–Grounded Commonsense for Task Decomposition in LLM Robot Planning

Abstract

Robotic planning breaks down when commonsense reasoning is required to resolve linguistic ambiguity and to interpret objects correctly. To address this, we present ConceptBot, a modular planning framework that integrates large language models with knowledge graphs to produce feasible, risk-aware plans while jointly disambiguating instructions and grounding object semantics.

ConceptBot comprises three components: (i) an Object Properties Extraction (OPE) module that augments scene understanding with semantic concepts from ConceptNet; (ii) a User Request Processing (URP) module that resolves ambiguities and structures free-form instructions; and (iii) a Planner that synthesizes context-aware, feasible pick-and-place policies. Evaluations in simulation and on real-world setups show consistent gains over prior LLM-based planners—for example, +56 percentage points on implicit tasks (87% vs. 31% for SayCan) and +61 points on risk-aware tasks (76% vs. 15%)—and an overall score of 80% on SafeAgentBench. These improvements translate to more reliable performance in unstructured environments without domain-specific training.

ConceptBot Overview

Object Properties Extraction (OPE)

The OPE module extracts relevant attributes from objects in the environment using ViLD or YOLO for detection and ConceptNet for contextual relationships. It identifies properties such as fragility, stability, and dangerous to enhance planning.

User Request Processing (URP)

The URP module interprets natural language instructions, extracting keywords and querying ConceptNet for semantic relationships. This process eliminates ambiguities and generates structured, actionable commands aligned with the robot’s capabilities.

Planner

The Planner integrates information from OPE and URP to produce pick-and-place policies. By combining LLM scoring with affordance evaluations, it ensures safe, feasible, and contextually relevant actions.