TAC21: Neuromorphic tactile sensing for physical interaction and understanding

Goal

We will exploit neuromorphic event-driven encoding, biologically inspired computation and learning, spiking neural networks and their supporting hardware processors for the physical interaction of artificial agents with the external world. Tactile and proprioceptive sensing are paramount to perceive the physical characteristics of objects such as softness, weight, shape, texture, etc. It helps build a solid knowledge about the objects the robot has to interact with and plan adequate control strategies for grip control, manipulation and tactile exploration.

Tactile exploration

Leaders: C. Bartolozzi, E. Chicca, F. Zenke, R. Benosman, B. Tee, E. Donati

This project focuses on tactile exploration for objects classification. We will use the iCub robot (both in simulation and remotely, with support of the IIT group) to perform exploratory actions of increasing complexity. The robot will slide its fingertips on different objects, starting from planar shapes up to 3D complex objects. Tactile information will be used to guide the exploratory movements of the robot. Contact detection and stimuli pattern and orientation will be used to find and follow the contour of objects. The recognition of the object at hand will be performed by integrating information from multiple sensory modalities over time: tactile sensor for texture and orientation, fingertip trajectories and movements from proprioception (motor's encoders, position sensors in the fingers, F/T sensors in the joints, etc.). Eye movements can complement the tactile and proprioceptive information, studying the effect of sensor fusion on the classification.

Tactile exploration for object recognition will be the context to study the use of event-driven sensors and Spiking Neural Networks for decision making to guide the exploratory actions of the robot, multisensor fusion, spatio-temporal pattern recognition

Check out the template code we wrote for a quick start:

Braille recognition: https://github.com/event-driven-robotics/telluride_tac21_braille and

https://github.com/dexrob/Telluride2021-BrailleLetter-NUSkin - the first with recordings for the IIT iCub skin and the second from the NUS skin

Contour following: https://github.com/event-driven-robotics/telluride_tac21_contour

Docker Image : https://hub.docker.com/repository/docker/eventdrivenrobotics/telluride

Video tutorial on how to install and run the code is below!

Neuromorphic tactile circuit design

Leaders: C. Bartolozzi, E. Chicca, B. Tee

This project focuses on the design and characterisation of neuromorphic tactile sensing using CMOS technology and different types of tactile transducers. Spice simulations will be used together with raw data from existing transducers (piezoelectric, capacitive, resistive, etc.) for the circuit development and characterisation.