Visual-tactile Fusion for Transparent Object Grasping in Complex Backgrounds

Authors: Shoujie Li, Haixin Yu, Wenbo Ding, Houde Liu, Linqi Ye, Chongkun Xia, Xueqian Wang,  Xiao-Ping Zhang

Abstract: The accurate detection and grasping of transparent objects are challenging but of significance to robots. In this article, a visual-tactile fusion framework for transparent object grasping in complex backgrounds is proposed, which synergizes the advantages of vision and touch, and greatly improves the grasping efficiency of transparent objects. First, we propose a multi-scene fully synthetic grasping dataset named SimTrans12K together with a Gaussian-Mask annotation method. Next, based on the TaTa gripper, we propose a grasping network named transparent object grasping convolutional neural network (TGCNN) for grasping position detection, which shows good performance in both synthetic and real scenes. Inspired by human grasping, a tactile calibration algorithm and  a visual-tactile fusion method are designed, which improve the grasping success rate by 36.7% compared to direct grasping and the classification accuracy by 39.1%. Furthermore, the tactile height sensing (THS) module and the tactile position exploration (TPE) module are added to solve the problem of grasping transparent objects in irregular and visually undetectable scenes. Experimental results demonstrate the validity of the proposed framework.

Introduction: Inspired by the grasping behavior of humans shown in the following figure, where visual and tactile sensations are collaboratively working towards complicated tasks, a visual-tactile fusion-based framework using the TaTa gripper is proposed in this paper for transparent object grasping in complex backgrounds. Here the tactile sensation is utilized to compensate for the limitation of vision. In addition, the framework can be extended to cover more challenging scenes such as irregular backgrounds or even visually undetectable scenes. 

Thirdly, to realize transparent object grasping in complex backgrounds, we propose a visual-tactile fusion-based transparent object grasping framework with tactile calibration. Besides, we add the tactile height sensing (THS) module and the tactile position exploration (TPE) module to this framework, which can achieve transparent object grasping in stacking, overlapping, or even visually undetectable scenes. Those scenes are extremely difficult and there are only a few studies before.

Finally, to test the effectiveness of the proposed visual-tactile fusion framework, we carefully design several experiments to extensively compare the performance with several state-of-the-art baseline methods, which indicates the proposed method has a considerable performance improvement for transparent object grasping and classification. Moreover, we also test the proposed method in some highly difficult scenes such as stacking, overlapping, undulating, and dynamic underwater environments, which greatly extends the application areas of transparent object grasping.

Hardware

Methodology

Grasping Strategy

For the comparison of the effect of grasp position detection networks, we compare the currently more mainstream generative grasp networks such as GGCNN, Redmon, and GI-NNet. Since most of these networks are designed for parallel two-finger grippers based on RGBD input, we modified the output structure of the networks to enable them to operate on our proposed dataset. So we change their output from Qmap, width, and angle to position and radius, and change the input data to RGB.

In our experiment, we evaluate TGCNN in the following scenarios: (a) Image-wise evaluation with unseen backgrounds; (b) Object-wise evaluation with unseen objects; (c) Effect of Gaussian representation; (d) Multi-objects evaluation in cluttered scenes, respectively.

Exp. 2: Grasping Position Detection in Different Backgrounds

To verify the detection effectiveness of the transparent object grasping network in real scenes, we select 12 different backgrounds, including 6 color, 4 patterned, and 2 scenic ones. The 4,000 synthetic data of two transparent objects are selected as the training set and 110 real data of 6 transparent objects as the test set which contains about 600 labels.

Exp. 3: Grasping Position Detection under Different Brightness

Besides the background, the intensity of light is also an important factor affecting the detection of transparent objects. The 4,000 synthetic data of two transparent objects are selected as the training set, and 50 real data of 6 transparent objects at different brightness as the test set. We test the effect of lightness on transparent object grasping detection by changing the brightness, using a Lux meter to measure the light intensity.

Exp. 4: Grasping and Classification on Planes in Complex Backgrounds

Exp. 5: Transparent Fragment Grasping

Exp. 6: Grasping in Irregular Scenes

Besides transparent object grasping on a plane, it is more challenging to grasp transparent objects in irregular scenes, where the position and height of where the transparent objects are located are difficult to obtain. To solve this problem, we added the THS module based on the previous framework to obtain the height where the object is located by tactile. For the overlapping scene, we designed a case where the objects are completely touching together. In this case, since the texture of transparent objects will be consistent with the background, it is difficult to separate them without a priori and depth information, so we need to assume that there is a certain height difference between the objects in the scene, otherwise, it is difficult to separate the objects touching together.

Exp. 7: Grasping in Visually Undetectable Scenes

Although the addition of the THS module can solve the problem of transparent object grasping in irregular scenes, it is still difficult to grasp transparent objects in scenes with visual interference, such as high-dynamic underwater environments. Still, the addition of the TPE module can solve this problem. To test the effectiveness of the modules, we conducted high-dynamic underwater transparent object grasping experiments , using tactile to search for transparent objects.

Conclusion

To solve the challenging problem of detecting, classifying, and grasping transparent objects, a visual-tactile fusion framework based on the synthetic dataset is proposed. First, we use the Blender simulation engine to render synthetic datasets instead of manually annotating datasets, solving the problem of difficult annotation of transparent object datasets. Besides, we use Gaussian-Mask instead of the traditional binarized annotation to make the generation of the grasping position more accurate. To achieve grasping position detection for transparent objects, an algorithm named TGCNN is proposed and multiple comparative experiments are conducted, which show that the algorithm can achieve good detection under different backgrounds and lighting conditions even when trained with only synthetic datasets.  Considering the grasping difficulty caused by the smooth surface of transparent objects, we propose a tactile calibration method combined with the soft gripper TaTa to improve the grasping success rate by adjusting the grasping position with tactile information. The method improves the grasping success rate by 36.7% compared to direct grasping. Furthermore, to solve the classification problem of transparent objects in complex scenes, a transparent object classification method based on visual-tactile fusion is proposed, which improves the accuracy by 39.1% compared to the classification with vision alone. Besides, to achieve transparent object grasping in irregular and visually undetectable scenes, we propose the THS and TPE modules, which can compensate for the problem of transparent object grasping in the absence of visual information. Extensive experiments are designed systematically and the results verify the effectiveness of the proposed framework in various complex scenarios, including stacking, overlapping, undulating, sand, underwater scenes, etc. We believe that the proposed framework can also be applied to object detection in low-visibility environments such as smoke and murky underwater, where tactile perception can compensate for the shortcomings of visual detection and improve classification accuracy by using visual-tactile fusion.

Additional Materials

We further compare the effect of depth information for the detection of undulating and underwater scenes, as shown in the following video . Firstly, for undulating scenes, reflections and folds may affect the detection of the depth cameras, and it is difficult to accurately detect the depth information of transparent objects with depth cameras due to their optical properties. Secondly, for underwater scenes, with the depth camera, transparent objects are almost integrated with water, and there will be a lot of reflections on the surface of the water under the illumination of light, which brings great influence to the detection. Therefore, in addition to the lack of images of transparent objects themselves, the interference of the environmental background is a tricky problem to solve, which is one of the reasons why we do not use depth images but use RGB images to achieve transparent grasping position detection in complex backgrounds.

Please see the paper for more details