Hanbit Oh*, Andrea M. Salcedo-Vazquez, Ixchel G. Ramirez-Alpizar, and Yukiyasu Domae
*: Corresponding author
2025 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) accepted
arXiv: [here] | code: [comming soon]
Imitation learning (IL) aims to enable robots to perform tasks autonomously by observing a few human demonstrations. Recently, a variant of IL, called In-Context IL, utilized off-the-shelf large language models (LLMs) as instant policies that understand the context from a few given demonstrations to perform a new task, rather than explicitly updating network models with large-scale demonstrations. However, its reliability in the robotics domain is undermined by hallucination issues such as LLM-based instant policy, which occasionally generates poor trajectories that deviate from the given demonstrations. To alleviate this problem, we propose a new robust in-context imitation learning algorithm called the robust instant policy (RIP), which utilizes a Student’s t-regression model to be robust against the hallucinated trajectories of instant policies to allow reliable trajectory generation. Specifically, RIP generates several candidate robot trajectories to complete a given task from an LLM and aggregates them using the Student's t-distribution, which is beneficial for ignoring outliers (i.e., hallucinations); thereby, a robust trajectory against hallucinations is generated. Our experiments, conducted in both simulated and real-world environments, show that RIP significantly outperforms state-of-the-art IL methods, with at least 26% improvement in task success rates, particularly in low-data scenarios for everyday tasks.
Fig1. Overview of RIP in a Banana Picking Task: Input of the initial image and output of the robotic gripper's trajectory are collected through human demonstrations. From the demonstration dataset, contextual text data is tokenized: the visual 3D keypoints with semantic and geometric similarities are extracted from image inputs, and the action trajectory consists of a 3D triplet representing the gripper's posture. The LLM-based instant policy is fed the context data and new image keypoints, sampling the action trajectories multiple times. Using the Student's-t regression model, a robust trajectory is captured from the set of sampled action trajectories, and the robot performs the task following that reliable trajectory.
Our key contributions of this paper are as follows:
We develop a novel method to filter out hallucinations from trajectories of LLM-based instant policy by using the Student's-t regression model.
We present Robust Instant Policy (RIP), a novel robust in-context imitation learning algorithm (Fig 1) that can instantly obtain a robust LLM-based robot policy with just a few human demonstrations.
We validate our method (RIP) in both simulated and real-world environments, show that RIP significantly outperforms state-of-the-art imitation learning methods by at least 26%, specifically in low-data scenarios for everyday tasks.
To evaluate RIP, as shown in Fig. 3, a set of ten daily tasks is defined for simulated and real-world environments. These tasks involve a robot interacting with a set of everyday household objects as below:
Fig.2: Environments and Tasks
Diffusion Policy (DP) [Chi et al.]: A state-of-the-art IL algorithm that has demonstrated superior learning efficiency and generality compared to several prior IL methods.
Keypoint Action Tokens (KAT) [Di Palo et al.]: A state-of-the-art in-context IL algorithm introduces an LLM as an instant policy by introducing the keypoint-based observation-action space described in §III-A.
KAT-DP [Di Palo et al.]: An IL algorithm that combines KAT and DP on [Di Palo et al.]. Utilizing the keypoint representation of KAT for the observation-action space instead of the raw image input from the original DP resulted in performance comparable to KAT [Di Palo et al.].
RIP with Gaussian (RIPGauss): A baseline for ablation studies in RIP; using a normal Gaussian distribution instead of the Student’s-t distribution.
Qualitative analysis comparing trajectories generated by LLM-based instant policies (e.g., RIP, RIPGauss, KAT) on push cube simulation tasks.
Success condition: at the end of the episode, the cube is placed on the goal position
Demo
RIPGauss Failure
KAT Failure
RIP (Ours) Success
Success condition: at the end of the episode, the banana is picked up
Demo
RIPGauss Failure
KAT Failure
RIP (Ours) Success
Success condition: at the end of the episode, the drawer is opened
Demo
RIPGauss Failure
KAT Failure
RIP (Ours) Success
Success condition: at the end of the episode, the cabinet is closed completely
Demo
RIPGauss Failure
KAT Failure
RIP (Ours) Success
Success condition: at the end of the episode, the faucet is rotated
Demo
RIPGauss Failure
KAT Failure
RIP (Ours) Success
Success condition: at the end of the episode, the banana is picked up
Demo
KAT Failure
RIPGauss Failure
RIP (Ours) Success
Success condition: at the end of the episode, the hanger is hung on the stand
Demo
RIPGauss Failure
KAT Failure
RIP (Ours) Success
Success condition: at the end of the episode, the cup is vertically placed on the table
Demo
RIPGauss Failure
KAT Failure
RIP (Ours) Success
Success condition: at the end of the episode, the dust (cubes) is on the dust pan
Demo
RIPGauss Failure
KAT Failure
RIP (Ours) Success
RIP (Ours) Success
RIP (Ours) Success
Success condition: at the end of the episode, the bottle is in the bowl
Demo
RIPGauss Failure
KAT Failure
RIP (Ours) Success
RIP (Ours) Success
RIP (Ours) Success
The results of the quantitative analysis comparing the policy success rates of each method are presented as below:
Result 1: Comparing the success rate with 10 demonstrations between RIP and baseline methods for each task. The success rate of each method, except KAT in simulation, is measured over 10 test executions. For KAT in simulation, all five trajectories generated for RIP from one seed are tested, i.e., 50 test executions. This result shows that our method, RIP, significantly improves performance for KAT, with a 26% increase, and shows significantly better results than other state-of-the-art imitation learning methods across all tasks.
Result 2: Comparing the success rate of RIP and baselines regarding the number of demonstrations. All results represent the mean (line) and standard deviation (shaded) of the success rate on all the simulation tasks. The success rate of each method, except KAT, is measured over 10 test executions. For KAT, all five trajectories generated for RIP from one seed are tested, i.e., 50 test executions. This result shows that our approach (RIP) consistently demonstrates a superior success probability across all demonstration amounts.
To investigate this question, we conducted a quantitative analysis of three key factors in RIP design: the number of queries (Q), the degree of freedom (ν), and down-sampling methods. The results are as below:
RIP has three main design elements (Q, ν, down-sampling). (a), (b): Success rate comparison of RIP for different Q = {2, 3, 5, 10}, v = {1.25, 1.5, ∞}, where ∞ is the same as RIPGasuss. The results present the mean and standard deviation of the success rates obtained by testing the RIP with 20 demonstrations 10 times for each simulated task. (c): Success rate comparison of RIP without and with grasping-action-based down-sampling, represented as Normal and g-based, respectively. The results present the mean and standard deviation of the success rates obtained by testing the RIP with 10 demonstrations 10 times for each task.