Kelin Li¹², Shubham M Wagh², Nitish Sharma², Saksham Bhadani², Wei Chen¹, Chang Liu², and Petar Kormushev¹
1. Robot Intelligence Lab, Imperial College London 2. Extend Robotics
Introduction
Traditional robotic manipulation methods often involve exploring the entire manipulation space, which can be inefficient and time-consuming. An alternative approach, imitation learning (Learning from Demonstration, LfD), enables robots to learn tasks by observing expert demonstrations, allowing for generalization to new scenarios. This method is more efficient as it extracts key information from expert behavior and environment interactions. However, collecting quality demonstrations remains a challenge. To address this, the study proposes an immersive VR-based teleoperation setup for gathering demonstrations. The system uses an xArm7 robot with an Inspire robot hand, controlled via a Meta Quest 3 headset and SenseGlove, providing real-time haptic feedback. The proposed Haptic-ACT framework integrates RGB images, joint positions, and fingertip contact forces to enhance learning, particularly for soft object manipulation.
Abstract
Robotic manipulation is essential for the widespread adoption of robots in industrial and home settings and has long been a focus within the robotics community. Advances in artificial intelligence have introduced promising learning-based methods to address this challenge, with imitation learning emerging as particularly effective. However, efficiently acquiring high-quality demonstrations remains a challenge. In this work, we introduce an immersive VR-based teleoperation setup designed to collect demonstrations from a remote human user. We also propose an imitation learning framework called Haptic Action Chunking with Transformers (Haptic-ACT). To evaluate the platform, we conducted a pick-and-place task and collected 50 demonstration episodes. Results indicate that the immersive VR platform significantly reduces demonstrator fingertip forces compared to systems without haptic feedback, enabling more delicate manipulation. Additionally, evaluations of the Haptic-ACT framework in both the MuJoCo simulator and on a real robot demonstrate its effectiveness in teaching robots more compliant manipulation compared to the original ACT.
Haptic-ACT: overview
Communication and feedback within the immersive VR-based teleoperation system involve capturing the user's hand position and orientation with the Meta Quest 3. A digital twin is employed to calculate inverse kinematics for the real robot arm. Finger joint positions are captured using a SenseGlove and mapped to desired positions for the real robot hand. All commands are published through ROS, and motor values from the real robot hand are translated into fingertip forces, which are then applied to the user via the SenseGlove.
Flowchart of the proposed Haptic-ACT. The observations include RGB images from two cameras, the robot's joint positions, and the fingertip forces of the hand. Note that the transformer encoder (CVAE encoder) operates only during the training phase to compute the style variable for the transformer encoder (CVAE decoder). During the inference phase, the style variable is fixed at 0.
Experiments Setup
We conducted experiments to evaluate the proposed VR-based platform and Haptic-ACT framework using both the MuJoCo simulator and a real robot system. In the simulator, we modeled an Xarm7 robot arm with a dexterous hand to perform a pick-and-place task. We trained policies using 50 successful episodes of a scripted policy, comparing ACT and Haptic-ACT frameworks. Training was done with an NVIDIA RTX3070ti GPU.
For the real-world experiments, we used the VR-based teleoperation system and performed the same task with human demonstrators, under two haptic conditions: with and without SenseGlove. We collected 50 successful episodes for each condition and trained policies using an NVIDIA RTX4070 GPU. Results were compared across ACT, Haptic-ACT, and original demonstrations.
Simulation Dataset Collection
Real World Dataset Collection
video_1.mp4video_2.mp4Real World Experiment Trials
video_3.mp4video_4.mp4Bonus Videos
video_5.mp4Detailed Presentation Video
A detailed explanation of our method, results and analysis, and more experiment trails can be found in this video.
ICRA25_1482_VI_i.mp4