Realistic physics engines play a crucial role for learning to manipulate deformable objects such as garments in simulation. By doing so, researchers can circumvent challenges such as sensing the deformation of the object in the real-world. In spite of the extensive use of simulations for this task, few works have evaluated the reality gap between deformable object simulators and real-world data.
We present a benchmark dataset to evaluate the sim-to-real gap in cloth manipulation. The dataset is collected by performing a dynamic as well as a quasi-static cloth manipulation task involving contact with a rigid table. We use the dataset to evaluate the reality gap, computational time, and simulation stability of four popular deformable object simulators: MuJoCo, Bullet, Flex, and SOFA. Additionally, we discuss the benefits and drawbacks of each simulator.
Record Dataset
Pre-process RGB-D data
Measure the Reality Gap in Simulated Environments
Content
We record our dataset using three rags from the public household cloth object set.
Our benchmark consists of the following:
a real-world dataset composed by point-clouds and depth images at each timestep of the cloth manipulation, using three garments with different material properties,
two cloth manipulation tasks, a dynamic task and a quasi-static task, simulated in four simulation engines,
metrics to evaluate the sim-to-real gap of the simulated environment along with the stability of the simulators.
We evaluate our dataset in MuJoCo 3.1.1, Bullet 3.26, Flex 1.2, and SOFA 23.06.
Prior to evaluating each simulation engine and cloth, we perform system identification over the following parameters:
MuJoCo: edge damping, mass, poisson ratio, thickness and young modulus.
Bullet: deform bending stiffness, deform damping stiffness, deform elastic stiffness, friction
coefficient, mass and cloth scale.
Flex: stretch, bend, shear and mass.
SOFA: young modulus, poisson ratio, damping, stiffness and vertex mass.
Real-World Dataset
The Dataset
Link for the full dataset coming soon!
Dynamic Manipulation Task
Depth and RGB images from the Azure Kinect sensor - used for obtainin the point-cloud
Chequered Rag
Towel Rag
Linen Rag
Quasi-Static Manipulation Task
Chequered Rag
Towel Rag
Linen Rag
Dynamic Cloth Manipulation Sim-to-Real Gap
We use two metrics to evaluate the reality gap: Chamfer Distance (CD) and Hausdorff Distance (HD).
Chequered Rag
MuJoCo
CD = 0.068 ± 0.028
HD = 0.151 ± 0.034
Bullet
CD = 0.130 ± 0.067
HD = 0.241 ± 0.0
Flex
CD = 0.160 ± 0.134
HD = 0.270 ± 0.183
SOFA
CD = 0.067 ± 0.023
HD = 0.175 ± 0.050
Towel Rag
MuJoCo
CD = 0.079 ± 0.027
HD = 0.168 ± 0.032
Bullet
CD = 0.159 ± 0.093
HD = 0.208 ± 0.084
Flex
CD = 0.166 ± 0.127
HD = 0.276 ± 0.158
SOFA
CD = 0.075 ± 0.026
HD = 0.189 ± 0.070
Linen Rag
MuJoCo
CD = 0.069 ± 0.027
HD = 0.150 ± 0.029
Bullet
CD = 0.127 ± 0.058
HD = 0.247 ± 0.071
Flex
CD = 0.160 ± 0.128
HD = 0.281 ± 0.186
SOFA
CD = 0.061 ± 0.026
HD = 0.148 ± 0.067
Quasi-Static Cloth Manipulation Sim-to-Real Gap
Chequered Rag
MuJoCo
CD = 0.049 ± 0.008
HD = 0.160 ± 0.036
Bullet
CD = 0.093 ± 0.034
HD = 0.223 ± 0.074
Flex
CDf= 0.041 ± 0.005
HD = 0.137 ± 0.017
SOFA
CD = 0.071 ± 0.017
HD = 0.162 ± 0.015
Towel Rag
MuJoCo
CD = 0.057 ± 0.020
HD = 0.154 ± 0.028
Bullet
CD = 0.079 ± 0.042
HD = 0.223 ± 0.087
Flex
CD = 0.052 ± 0.011
HD = 0.136 ± 0.016
SOFA
CD = 0.061 ± 0.008
HD = 0.135 ± 0.012
Linen Rag
MuJoCo
CD = 0.058 ± 0.023
HD = 0.154 ± 0.047
Bullet
CD = 0.061 ± 0.014
HD = 0.158 ± 0.014
Flex
CD = 0.047 ± 0.010
HD = 0.119 ± 0.036
SOFA
CD = 0.044 ± 0.007
HD = 0.106 ± 0.023
Reality Gap Results
The dynamic task distances have the subscript d, CDd ; and the quasi-static ones have the subscript subscripct q, CDq.
The results for MuJoCo, Bullet, Flex and SOFA are depicted in the table below.
The table shows the results for 20 different random seeds over three different datasets for each of the cloths.
Team
Adrià Colome
Institut de Robòtica i Informàtica Industrial, CSIC-UPC
Spain
Citation
To cite this work, please use the following BibTex entry:
@article{blancomulero2024benchmarking,
author={Blanco-Mulero, David and Barbany, Oriol and Alcan, Gokhan and Colomé, Adrià and Torras, Carme and Kyrki, Ville},
journal={IEEE Robotics and Automation Letters},
title={Benchmarking the Sim-to-Real Gap in Cloth Manipulation},
year={2024},
volume={9},
number={3},
pages={2981-2988},
doi={10.1109/LRA.2024.3360814}
}