A Digital Twin-Driven Mixed Reality Framework for
Immersive Teleoperation with Haptic Rendering
Wen Fan*, Xiaoqing Guo*, Enyang Feng*, Jialin Lin*, Yuanyi Wang, Jiaming Liang, Martin Garrad, Jonathan Rossiter, Zhengyou Zhang, Nathan Lepora, Lei Wei, Dandan Zhang
Abstract
Teleoperation empowers human operators to remotely control robots, which has widely contributed to applications such as telesurgery, industrial inspection, and remote manipulation in hazardous or contaminated environments. Consequently, the design of intuitive and ergonomic control interfaces for teleoperation becomes crucial. The rapid advancement of Mixed Reality (MR) in recent years has yielded tangible benefits in human-robot interaction. MR provides an immersive interaction platform with robots, effectively reducing the mental and physical workload of operators during teleoperation. Additionally, the incorporation of haptic rendering, including both kinaesthetic and tactile rendering, could further amplify the intuitiveness and efficiency of MR-based immersive teleoperation. In this study, we develop an immersive, bilateral teleoperation system. This system integrates Digital Twin-driven Mixed Reality (DTMR) manipulation with haptic rendering. It comprises a commercial controller with a kinaesthetic rendering feature and a new, wearable, cost-effective tactile rendering interface, called the Soft Pneumatic Tactile Array (SPTA). We perform two user studies to assess the effectiveness of our proposed system. These studies include a performance evaluation of the key components within the DTMR framework and a quantitative assessment of the newly developed SPTA device. The results demonstrate a qualitative and quantitative enhancement in both human-robot interaction and teleoperation performance.
Motivation
Teleoperation is widely used in many applications. Traditional teleoperation systems can provide visual feedback but usually lack tactile and force information.
During teleoperation, the ergonomics and intuitiveness of the motion-tracking devices are essential for operators to control the remote robot with high efficiency. While, the Mixed-Reality (MR) and Digital-Twins (DT) technology allow operators to interact with both physical and virtual objects in real-time, reducing the latency of visual feedback during control.
Digital Twin-Driven Mixed Reality
For the upper reason, a bilateral teleoperation system is desirable for the operator who manipulates remote objects and receives force/tactile feedback through the control interface.
So in this study, we develop an immersive teleoperation system, which integrates Digital Twin-driven Mixed Reality (DTMR) with haptic rendering. It comprises a commercial controller with a kinaesthetic rendering feature and a wearable, cost-effective tactile rendering interface, called Soft Pneumatic Tactile Array (SPTA).
Soft Pneumatic Tactile Array
SPTA is made up of two layers of soft film, forming a two-dimensional pattern of tactile actuators which serves as expandable air pockets. The number of expandable air pockets can be tailored t meet the demands of various applications, and the chamber diameter of each air pocket can also be customized. By activating different numbers of actuators, we can convey varying levels of tactile rendering intensity. By receiving the tactile information from the force sensor, the microcontroller commands the multi-channel relays to drive the pumps and valves, providing regulated pressure to airbags of SPTA through air tubes.
User study based on an Object Transportation Task
This user study aims to make comparisons between DTMR performance of whether the SPTA device is disabled or enabled. Our wearable SPTA device offers the flexibility to be attached to arbitrary fingers by user preference. Participants were asked to conduct a multiple-object transportation task. The experiment target is to transport three objects from the plate to the target empty box through DTMR manipulation.
Result Analysis
Referring to the evaluation metric of enhancement, the performance of task execution is much improved when using the SPTA device. In summary, results in this user study indicated that human operators can finish the task with higher efficiency and better performance when using tactile rendering, since it provides touch information with the target object as the supplementary feature beyond the visual display.
Questions?
Contact [ye21623@bristol.ac.uk, xq.guo@bristol.ac.uk ]
[Bristol Robotics Lab, University of Bristol, UK]