TacFlex: Multi-Mode Tactile Imprints Simulation for Visuotactile Sensors with Coating Patterns

Chaofan Zhang, Shaowei Cui, Jingyi Hu, Tianyu Jiang, Tiandong Zhang, Rui Wang, and Shuo Wang

Institute of Automation, Chinese Academy of Sciences

School of Artifificial Intelligence, University of Chinese Academy of Sciences

Abstract

Visuotactile sensors could provide rich contact information for robots. However, how to build a high-fidelity visuotactile simulator that supports multi-mode tactile imprints and various sensor configurations remains a challenging problem. In this paper, we present TacFlex, a flexible simulator for visuotactile sensors, which physically simulates the elastomer deformation using Finite Element Methods (FEM), and focuses on linking the deformed elastomer mesh to diverse tactile imprints, including tactile images with arbitrary coating patterns and tactile 3D point clouds. We further propose a ray tracing-based rectification method to deal with multi-medium refraction effects to make the simulated tactile images more realistic. Extensive experiments are conducted to demonstrate the effectiveness of TacFlex on several visuotactile sensors. Furthermore, we explore the Sim2Real performance of different tactile imprints provided by TacFlex in tactile perception and manipulation tasks, such as cylindrical object pose estimation and peg-in-hole. The perception/policy models trained in simulation are successfully deployed in real world. Finally, we present the outlook on the potential of TacFlex in robot learning.

TacFlex Features

Supporting multi-type visuotactile sensors, such as GelSight, GelStereo, and so on.
Integrating multi-mode tactile imprints, such as tactile images, marker motions, tactile 3D point clouds.
FEM-driven, supporting multiple FEM simulators.

TacFlex Pipeline

The TacFlex pipeline for visuotactile simulation is illustrated below. Deformation simulation is responsible for physically simulating the elastomer behaviors caused by sensor-object interactions and outputs the deformed mesh. TacFlex focuses on linking the deformed elastomer mesh to different tactile imprints, including tactile 3D point clouds and tactile images.

TacFlex Simulation Demonstrations

Non-physics-based or FEM-based？

Non-physics-based: only rely on the object's shape and pose, and cannot model the elastomer stretching.
FEM-based: simulate the elastic and contact dynamics, and could simulate the elastomer stretching caused by object sliding on the sensor surface.

Tacto_tacflex_gelsight_clip.mp4

TacFlex simulation for various visuotactile sensors

gelstereo_2.0_pressing_compressed.mp4

GelStereo 2.0 sensor

（flat sensor surface, marker pattern）

gelstereo_palm2.0_compressed.mp4

GelStereo Palm2.0 sensor

（curved sensor surface, marker pattern）

gelsight_slide.mp4

GelSight Mini sensor

（RGB lighting, marker pattern）

TacFlex simulation for sensors with different coating patterns

checker_sim_real_1_compressed.mp4

Checkerboard pattern

random_sim_real_1_compressed.mp4

Random color pattern

TacFlex simulation under complex contact states

gelstereo_2.0_sliding_compressed.mp4

Sim2Real Applications of TacFlex

Cylindrical object pose estimation

We apply a cylindrical object pose estimation task to show how TacFlex can be applied to learning-based tactile perception. A pose estimation network is trained using a dataset generated in TacFlex simulation. The dataset contains tactile image and object pose pairs. Then, this network realizes zero-shot Sim2Real transfer. We have evaluated the pose estimation errors on real sensors and carried out a tube placing task (success rate: 95.8%) by estimating the tube pose once.

tube_insertion_compressed.mp4

Peg-in-hole task

We carry out a peg-in-hole task to show how TacFlex can be applied to skill learning for robot manipulation tasks. We employ the tactile 3D point clouds of GelStereo 2.0 sensors as the observation and feed the sequences of marker displacements into the policy network. The insertion policy is trained in the simulation with the principle of behavior cloning, and then deployed on a real robot. The training data is collected only using the peg with 2.0 mm clearance, and the policy is tested using pegs with different clearances, such as 0.6 mm clearance.

peg_hole_sim_real_compressed.mp4

peg_hole_real_compressed.mp4

Supplementary Video

video_final_compressed.mp4

Page updated

Google Sites

Report abuse