Yijiong Lin, Mauro Comi, Alex Church, Dandan Zhang, Nathan F. Lepora
Department of Engineering Mathematics and Bristol Robotics Laboratory, University of Bristol, Bristol, U.K.
Email: yijiong.lin@bristol.ac.uk
Overview
High-resolution tactile sensing can provide accurate information about local contact in contact-rich robotic tasks. However, the deployment of such tasks in unstructured environments remains under-investigated. To improve the robustness of tactile robot control in unstructured environments, we propose and study a new concept: tactile saliency for robot touch, inspired by the human touch attention mechanism from neuroscience and the visual saliency prediction problem from computer vision. In analogy to visual saliency, this concept involves identifying key information in tactile images captured by a tactile sensor.
Fig. 1: Visual saliency vs tactile saliency: (a) an example of visual saliency map (right) and its source visual image (mid) from SALICON dataset; (b) an example of tactile saliency map (right) and its source real tactile image (mid) obtained by a TacTip (left) contacting a target edge (white cylinder) and a distractor (pink toy) in an edge-following task in a cluttered scene.
Motivation and Methods:
While visual saliency datasets are commonly annotated by humans, manually labelling tactile images is challenging due to their counterintuitive patterns. To address this challenge, we propose a novel approach comprised of three interrelated networks: 1) a Contact Depth Network (ConDepNet), which generates a contact depth map to localize deformation in a real tactile image that contains target and noise features; 2) a Tactile Saliency Network (TacSalNet), which predicts a tactile saliency map to describe the target areas for an input contact depth map; 3) and a Tactile Noise Generator (TacNGen), which generates noise features to train the TacSalNet. Experimental results in contact pose estimation and edge-following in the presence of distractors showcase the accurate prediction of target features from real tactile images. Overall, our tactile saliency prediction approach gives robust sim-to-real tactile control in environments with unknown distractors.
Fig. 2: Overview of the proposed 3-stage approach for tactile saliency prediction: (a) ConDepNet training for generating a contact depth map to localize deformation in a real tactile image using a dataset of paired real and simulated tactile images. (b) TacSalNet training to predict a tactile saliency map to capture a desired target feature for an input contact depth map with noise. (c) TacNGen training for generating noise features to train the TacSalNet.
Tactile Saliency Prediction for the Contact Pose Prediction:
In this experiment, we evaluate the performance of tactile saliency prediction in improving contact pose prediction accuracy in the presence of distractors of various shapes.
Vid. 1: Tactile Saliency Prediction for Contact Pose Estimation in the presence of various distractors
(Click to expand the result figure)
Fig. 3: Real-world evaluation of tactile saliency prediction based on the tactile PoseNet prediction accuracy on the target feature (edge) distracted by various everyday objects: (a) the distraction setup with everyday objects of various shapes; (b) real tactile images of the local contact paired with related contact depth map and tactile saliency map; (c) tactile PoseNet prediction accuracy on 2D edge pose without and (d) with tactile saliency prediction.
Tactile Saliency Prediction for the Edge-Following Task:
In this experiment, we investigate the performance of the tactile saliency prediction in improving the robustness of different tactile control methods during an edge-following task that involves distractors.
Fig. 4: The diagram of two types of sim-to-real tactile control methods augmented with tactile saliency prediction for the edge-following task with unexpected distractors. The orange arrows denote the pipeline of tactile image processing, the blue and purple arrows denote the pipelines of the pose-based PID control and image-based deep RL policy respectively. The network in green is trained with real-world and simulated data, while the ones in red are obtained solely in simulation and applied to the real world without fine-tuning.
More interesting applications:
In these qualitative experiments, we further demonstrate the generalizability and scalability of our tactile saliency prediction method in the real-world edge-following task without any further finetuning.
@article{lin2023attention,
title={Attention for Robot Touch: Tactile Saliency Prediction for Robust Sim-to-Real Tactile Control},
author={Lin, Yijiong and Comi, Mauro and Church, Alex and Zhang, Dandan and Lepora, Nathan F},
journal={arXiv preprint arXiv:2307.14510},
year={2023}
}
We introduce a dual-arm tactile robotic system (Bi-Touch) based on the Tactile Gym 2.0 setup that integrates two affordable industrial-level robot arms with low-cost high-resolution tactile sensors. A suite of bimanual manipulation tasks tailored toward tactile feedback is developed and open-sourced.
High-resolution optical tactile sensors are increasingly used in robotic learning environments due to their ability to capture large amounts of data directly relating to agent-environment interaction. We extend the Tactile Gym simulator to include three new optical tactile sensors (TacTip, DIGIT, and DigiTac) of the two most popular types, Gelsight-style (image-shading based) and TacTip-style (marker-based). We demonstrate that a single sim-to-real approach can be used with these three different sensors to achieve strong real-world performance despite the significant differences between real tactile images. Additionally, we lower the barrier of entry to the proposed tasks by adapting them to an inexpensive 4-DoF robot arm, further enabling the dissemination of this benchmark.
A desktop robot is performing a surface-following task learned from simulation.
DIGIT
DigiTac
TacTip
