Guanqun Cao a Jiaqi Jiang b Danushka Bollegala a Min Li c Shan Luo b
a. Department of Computer Science, University of Liverpool, Liverpool, L69 3BX, United Kingdom
b. Department of Engineering, King’s College London, London, WC2R 2LS, United Kingdom
c. School of Mechanical Engineering, Xi’an Jiaotong University, Xi’an, 710049, China
Abstract
Tactile sensing plays an irreplaceable role in robotic material recognition. It enables robots to distinguish material properties such as their local geometry and textures, especially for materials like textiles. However, most tactile recognition methods can only classify known materials that have been touched and trained with tactile data, yet cannot classify unknown materials that are not trained with tactile data. To solve this problem, we propose a tactile zero-shot learning framework to recognise unknown materials when they are touched for the first time without requiring training tactile samples. The visual modality, providing tactile cues from sight, and semantic attributes, giving high-level characteristics, are combined together to bridge the gap between touched classes and untouched classes. A generative model is learnt to synthesise tactile features according to corresponding visual images and semantic embeddings, and then a classifier can be trained using the synthesised tactile features of untouched materials for zero-shot recognition. Extensive experiments demonstrate that our proposed multimodal generative model can achieve a high recognition accuracy of 83.06% in classifying materials that were not touched before. Our proposed method for zero-shot tactile recognition has a potential application in robotic textile sorting and recycling industry.
Paper
Multimodal Zero-Shot Learning for Tactile Texture Recognition
Guanqun Cao, Jiaqi Jiang , Danushka Bollegala, Min Li, and Shan Luo
Accepted at Robotics and Autonomous Systems
Dataset
Our dataset is collected from 50 different fabrics, and some examples are illustrated in the figure below.
Available at: Link to dataset
Application
By using the proposed method, the robot can use tactile sensing to recognize the unknown materials when touching them for the first time.
The fabric can still be recognized even when the appearance is changed (e.g., the fabric is folded).