SafePlan: Leveraging Formal Logic and Chain-of-Thought Reasoning for Enhanced Safety in LLM-based Robotic Task Planning

Abstract

Robotics researchers increasingly leverage large language models (LLM) in robotics systems, using them as interfaces to receive task commands, generate task plans, form team coalitions, and allocate tasks among multi-robot and human agents. However, despite their benefits, the growing adoption of LLM in robotics has raised several safety concerns, particularly regarding executing malicious or unsafe natural language prompts. In addition, ensuring that task plans, team formation, and task allocation outputs from LLMs are adequately examined, refined, or rejected is crucial for maintaining system integrity. 

In this paper, we introduce SafePlan, a multi-component framework that combines formal logic and chain-of-thought reasoners for enhancing the safety of LLM-based robotics systems. Using the components of SafePlan, including Prompt Sanity COT Reasoner and Invariant, Precondition, and Postcondition COT reasoners, we examined the safety of natural language task prompts, task plans, and task allocation outputs generated by LLM-based robotic systems as means of investigating and enhancing system safety profile. Our results show that SafePlan outperforms baseline models by leading to 90.5% reduction in harmful task prompt acceptance while still maintaining reasonable acceptance of safe tasks. 

System Overview 

SafePlan comprises two main components that are deployed across three main stages. The first component and the first stage is the Prompt Sanity Check COT Reasoner which parses a robot task prompt into its essential elements (e.g., actions, entities, and resources) and evaluates them against multiple layers of safety criteria (societal, organizational, and individual) to intercept harmful or unsafe commands without requiring extensive fine-tuning or specialized datasets. The second component of SafePlan is the Invariant COT Reasoner, which employs Linear Temporal Logic (LTL) to formalize safety properties within a state-transition model, allowing for principled verification of invariants throughout task execution by generating invariants, preconditions, and postconditions for natural language robot task prompts. During the second stage, the output of the Invariant COT Reasoner is employed as a few-shot example to guide the robot system in code generation. While during the third stage, we further employ the Invariant COT Reasoner to verify and ensure that code output from the LLM's output aligns with the earlier invariants, preconditions, and postconditions. For instance, given a natural language task prompt to a robot "place this gun on the sofa," our framework first uses the Prompt Sanity COT reasoner to examine the impact of completing this task and ultimately responding with a task denial and removing the task away from the pipeline. However, for another task prompt "place this cup on the kitchen sink, " the Prompt Sanity Check COT Reasoner examines and approves the task as safe for the pipeline, then the Invariant COT Reasoner uses the task prompt to generate the invariant, preconditions, and postconditions that must hold true for completing this task. This output together with the original approved task prompt, robot skills, and objects recognized in the space are passed to an LLM as a few-shot prompt to generate code for completing the task if the combination of the object list, and robot skill, in addition to the goal of the task scales through the invariant checks. 

Experiments

We contribute a benchmark comprising over 600 expert-curated prompt-scene description pairs and conduct a simulated experimental study on AI2-THOR to demonstrate the efficacy of our approach within a robotics system pipeline.

Results

Paper

Link to paper: https://arxiv.org/pdf/2503.06892

BibTex

@article{obi2025safeplan,

  title={SafePlan: Leveraging Formal Logic and Chain-of-Thought Reasoning for Enhanced Safety in LLM-based Robotic Task Planning},

  author={Obi, Ike and Venkatesh, Vishnunandan LN and Wang, Weizheng and Wang, Ruiqi and Suh, Dayoon and Amosa, Temitope I and Jo, Wonse and Min, Byung-Cheol},

  journal={arXiv preprint arXiv:2503.06892},

  year={2025}

}