A Multi-modal Tactile Fingertip Design for Robotic Hands to Enhance Dexterous Manipulation

Zhuowei Xu, Zilin Si, Kevin Zhang, Oliver Kroemer, Zeynep Temel

Carnegie Mellon University

Abstract

Tactile sensing holds great promise for enhancing manipulation precision and versatility, but its adoption in robotic hands remains limited due to high sensor costs, manufacturing and integration challenges, and difficulties in extracting expressive and reliable information from signals. In this work, we present a low-cost, easy-to-make, adaptable, and compact fingertip design for robotic hands that integrates multi-modal tactile sensors. We use strain gauge sensors to capture static forces and a contact microphone sensor to measure high-frequency vibrations during contact. These tactile sensors are integrated into a compact design with a minimal sensor footprint, and all sensors are internal to the fingertip and therefore not susceptible to direct wear and tear from interactions. From sensor characterization, we show that strain gauge sensors provide repeatable 2D planar force measurements in the 0–5 N range and the contact microphone sensor has the capability to distinguish contact material properties. We apply our design to three dexterous manipulation tasks that range from zero to full visual occlusion. Given the expressiveness and reliability of tactile sensor readings, we show that different tactile sensing modalities can be used flexibly in different stages of manipulation, solely or together with visual observations to achieve improved task performance. For instance, we can precisely count and unstack a desired number of paper cups from a stack with 100% success rate which is hard to achieve with vision only.

Fingertip Design

Fingertip structure includes a rigid bone and a fingernail, and is covered by a soft skin, which mimics human fingertip’s mechanical structure. We leverage strain gauge sensors to emulate the role of slow adapting (SA) mechanoreceptors, responsible for detecting pressure, and a contact microphone sensor to replicate the function of fast adapting (FA) mechanoreceptors, to pick up contact vibrations and infer dynamic contact events such as slip

Fingertip Fabrication

The fabrication procedure and final prototype of the fingertip. (a) Fingertip fabrication: 1. Soldering the connector onto the PCB; 2. attaching the contact microphone sensor on the top of the 3D printed cap using super glue; 3. molding a soft skin for the cap with silicone (Mold Star 20T) by using two 3D-printed molds; 4. gluing four strain gauge sensors on each side of the square prism; 5. mounting the PCB with M2 screws below the fingertip base; 6. Assembling the cap and the base with an M2 screw; 7. wearing the soft skin on the cap. (b) A fully assembled fingertip prototype.

Readout Circuit

Readout circuit of the fingertip: the contact microphone sensor signals and the strain gauge sensor signals are collected through the fingertip PCB and carried by an FFC cable to another custom PCB. The strain gauge sensor measurements are amplified and digitized with an HX711 module on an Arduino Uno, while the vibrotactile signals are pre-amplified, digitized by a modified Maono USB sound card. Both are transmitted to the PC via USB.

Sensor Characterization

Tactile sensor characterization setup and results: (a) Front and side views of the setup for strain gauge sensor characterization. A fingertip sits on a 6D force/torque sensor that is rigidly fixed on the table. A UR5e robotic arm with a custom end effector (3 mm cylindrical indenter) is used to step and press on the fingertip from different directions and with incremental indentation depths to characterize 2D planar force sensing. (b) Front and side views of the setup for contact microphone sensor characterization. We use a custom end-effector with samples of seven materials attached to different faces. The UR5e arm is controlled to initiate sliding contacts between the fingernail and a sample to generate vibrotactile signals. (c) We show the linear correspondence between the strain gauge sensor measurements and the ground-truth force readings from 0 to 5 N at a fixed indentation depth across different directions. (d) We learn a material classification model by using vibrotactile measurements. We show the results on the test dataset.

video-sec1.mp4

Experiments

Overview

We evaluated the proposed tactile fingertips across three manipulation tasks: i) pinching fragile objects with fingertip force control, ii) counting and unstacking paper cups, and iii) detecting the material of hidden objects through shaking to guide subsequent manipulation. The first task examines whether fingertip tactile signals are reliable enough to serve directly as feedback for closing the control loop. The second investigates whether tactile sensing can serve as an alternative to vision under occlusion to improve manipulation performance. The third evaluates the sensitivity and robustness of tactile sensing for material recognition and its effectiveness in informing downstream manipulation.

Fragile object pinching with fingertip force control

video-sec2.mp4

Cup counting and unstacking

Cup counting and unstacking pipeline. (a) Start: hand aligned over a cup stack. (b) Moving down to slide through a stack of cup edges. (c) Locate the cup edges based on the vibrotactile features, and move the hand down to the target cup edgs location. (d) Close the finger and lift the cups. (e) Separate and unstack the cups. (f) Binarize raw vibrotactile signals to locate the moment of encountering cup edges during sliding, and synchronize with the robot height. (g) vision-based baseline: using text prompts and keyframes from the external camera to query cup counting with a GPT-5 agent. (h) We show vibrotactile-based approach outperforms the vision-based approach, especially with visual occlusion scenarios.

video-sec34.mp4

Hidden object detection through shaking

Experiment setup and pipeline for occluded in-box object material detection and box opening. (a) We offer two boxes, which are visually identical. One box has screws inside and another includes rubber bands, to represent rigid and compliant object. (b) The robot sequentially grasps a box and shake it; (c) The vibrotactile readings collected during shaking are used to predict the hidden material class with a learned model; (d) The robot is guided to select the box that contained the target material class and open it.

video-sec5.mp4

video-sec6.mp4

Page updated

Google Sites

Report abuse