TacEx: GelSight Tactile Simulation in Isaac Sim Combining Soft-Body and Visuotactile Simulators

Paper

Code

TacEx has been greatly improved since we presented our workshop paper.

It is easier to use, more flexible and has more features now.

Current Features Include:

GPU accelerated Tactile RGB simulation via Taxim's simulation approach
Marker Motion Simulation via FOTS
Integration of UIPC (instead of GIPC) for GPU accelerated incremental potential contact to simulate FEM soft bodies, rigid bodies, cloth, etc. in a penetration-free and robust manner
Marker Motion Simulation with FEM soft body based on the simulator used by the ManiSkill-ViTac challenge that leverages UIPC

New Demonstrations:

Adding Tactile Sensors to Environments

The GPU-Taxim based simulation can be easily added to existing Isaac Lab envs. For example, some contact-rich-manipulation env's with tactile sensors.

tactile_nutThread.mp4

tactile_gear.mp4

Ball Rolling RL env

We created a few RL environments which leverage our tactile simulation.

One of them is the Ball-Rolling-RL env.

The goal here is to move the ball to a random target position.

The speed and efficiency of the GPU-Taxim based simulation allow for (relatively) fast prototyping.

The video shows the training of a PPO agent, with proprioception + Tactile RGB images as observations. The training was done with 512 parallel env's.

tactile_rgb_5h_edit.mp4

UIPC for Tactile Simulation

We integrate the new UIPC framework to simulate the gelpad as a FEM soft body.

We can use the deformation of the FEM body to simulate the marker motion:

mani_skill_marker_sim.mp4

Integration of UIPC inside Isaac Sim

UIPC (libuipc) has a lot of examples. Here are some of them inside Isaac Sim.

bunnyXcloth.mp4

some_more_examples.mp4

Tactile Playground

We also have an playground to checkout/ test the tactile simulation output on the shapes from Gomes et al. For example the FOTS simulation:

fots_showcase.mp4

Simulating the Gelpad with Texture

This is another showcase for what you could do with the UIPC simulation. We simulate the gelpad via UIPC as a soft body and apply a GelSight-Mini texture on it.

textured_higher_res.mp4

While the deformation looks pretty nice, the image is very blurry.

This is most likely due to complex lighting and the render settings

(which we did not tweak at all for all the demonstrations here).

texture_based_without_light.mp4

This showcases the UIPC-Texture simulation with lights turned off.

The blurriness seems to be significantly reduced.

Old Demonstrations (from the Workshop-Paper version)

We run the experiments on an Ubuntu system with

an AMD Ryzen9 5950X 16-Core CPU,
32 GB RAM
an NVIDIA RTX 3080Ti GPU with 12 GB VRAM.

Ball Rolling Experiment

ball_rolling_rigid_tactile.mp4

rigid body with compliant contact

ball_rolling_soft.mp4

PhysX soft body gelpad

ball_rolling_gipc.mp4

GIPC based soft body simulation

(sensor case turned invisible for
better view of the soft body deformation)

Object Lifting

PhysX soft body gelpads for lifting rigid/soft objects

soft_lift_(speedup).mp4

lifting of rigid objects

(video is partially sped up)

the object cannot be lifted

softbody_quad.mp4

lifting of a soft object

during the grasp, the gelpads are crushed completely

the soft object is being lifted by the rigid bodies behind the gelpads
when only the gelpads are in contact, then the object falls down immeditately

GIPC gelpads for lifting (and squishing) soft objects

GIPC_LIFT.mp4

coarse tet meshes were used here for cube and gelpads

GIPC_LIFT_BALL.mp4

Beam Twisting

twist_custom_tet.mp4

gelpads were turned invisible for better view of the beam
at the end the tetrahedra mesh is shown (in white)

Page updated

Google Sites

Report abuse