Embedded Files
The in-hand object pose estimation is still a challenging problem in the robot community, especially for small objects. The emerging visuotactile sensors with high-resolution and high-precision 3D contact geometry sensing capability provide an approach using the Point Cloud Registration (PCR) method. The ground truth of in-hand object pose is difficult to obtain in the real world, while it is natural in the simulation environment. To this end, we create and release an in-hand object pose estimation dataset in the marker-level visuotactile simulation. This dataset contains 24000 samples for 20 objects, and aims at pose estimation of small parts and tools using tactile 3D point clouds on robot fingertips.
The dataset contains the following items:
The tactile 3D point clouds from two fingertips.
The ground-truth object poses.
The 3D model of the objects, including the STL file and point cloud file.
Data Collection
Data Collection
We choose 20 objects from industrial assembly scenarios and daily life, such as bolts of different types, cylinders, spanners, screwdrivers, knives, and keys. In the simulation, two GelStereo sensors are used as the gripper's fingertips to parallel grasp these objects in various poses. For ease of downstream manipulation tasks, such as screwing a bolt, cutting, and inserting, the handle part of the object is preferred to grasp.
The configurations for data collection on each object:
Sample 60 object poses randomly in the following ranges:
Contact depth: 0.5 mm, 1.0 mm, 1.5 mm, and 2.0 mm
Marker intervals on the sensor surface: 0.2 mm, 0.4 mm, 0.6 mm, 0.8 mm, and 1.0 mm
Download
Download
Click here to download the dataset.
Data Format
Data Format
- Ground-truth object pose
File path: ./pose ground truth/'object name'/*.txt
Each txt file contains the ground-truth pose in the form of (x, y, z, Rz, Rx, Ry). The transformation matrix (4×4) can be computed by python ./scripts/compute_gt_transformation_matrix.py
- Tactile 3D point clouds
File path: ./tactile pcd/'object name'/marker_**/depth_**/*.npy
The npy file shape is (4, N, 3), where N is the number of tactile markers on each sensor. Each data point contains the marker position in Cartesian space in the form of (x, y, z). The 4 channels indicate the tactile points on sensor 1, the tactile points on sensor 1 with no contact, the tactile points on sensor 2, and the tactile points on sensor 2 with no contact, respectively. Note that the differences between channel 0 (2) and 1 (3) are the displacements of tactile points.
Baseline: ICP for Pose Estimation
Baseline: ICP for Pose Estimation
- Demo
python ./scripts/icp_o3d_test.py
- Metrics
- Results and Analysis
The Iterative Closest Point (ICP) algorithm is employed as the baseline method for in-hand object pose estimation using tactile 3D point clouds. We test the estimation errors on each object, and provide some insights on PCR-based pose estimation.
Balance of salient point number and confidence
We test different contact depths and salient point extraction parameters for in-hand object pose estimation. For an object, increasing the contact depth will increase the number of salient points, and a large salient parameter provides points with high confidence. How to balance the point number and confidence or how to deal with the tactile points with different confidences is worth exploring.
The coverage degree of the tactile points on the object surface
Different from the point cloud from the vision sensor, tactile point clouds usually consist of several separate patches from multiple tactile sensors. We try to study the effects of patch number on pose estimation. In practice, we conduct ICP using tactile point clouds from one and two sensors, respectively. From the results in Table IV, we find that a larger number of patches is beneficial to improving the estimation accuracy, which implies that multi-fingered hands with tactile capability are superior in object pose estimation. Furthermore, the effect of the spatial distribution of tactile patches on the object surface is an interesting topic.
Point spatial resolution on the sensor surface
The spatial resolution of tactile points depends on the sensor in the real world. We test the object pose estimation using different point resolutions generated in the simulation, as shown in Table V. The experimental results indicate that tactile point clouds with high spatial resolution could help with object pose estimation. In the real world, the sparse-dense point cloud registration problem might need to be solved to improve the pose estimation accuracy.
Initial pose deviation for ICP algorithm
We divide the initial object pose deviation into 5 levels. From levels 1 to 5, the deviation increases. The performance of the ICP algorithm is sensitive to the initial pose, as shown in Table VI. The point cloud registration method, which is insensitive to the initial pose, needs to be investigated, or global registration could be employed to decrease the initial pose deviation. In addition, visual perception could be an approach for coarse pose estimation.
Page updated
Google Sites
Report abuse