TEXterity: Tactile Extrinsic deXterity
*Antonia Bronars1, *Sangwoon Kim1, Parag Patre2, and Alberto Rodriguez1
*Equal Contribution, 1MIT, 2Magna International Inc.
Abstract
We introduce a novel approach that combines tactile estimation and control for in-hand object manipulation. By integrating measurements from robot kinematics and an image-based tactile sensor, our framework estimates and tracks object pose while simultaneously generating motion plans to control the pose of a grasped object. This approach consists of a discrete pose estimator that uses the Viterbi decoding algorithm to find the most likely sequence of object poses in a coarsely discretized grid, and a continuous pose estimator-controller to refine the pose estimate and accurately manipulate the pose of the grasped object. Our method is tested on diverse objects and configurations, achieving desired manipulation objectives and outperforming single-shot methods in estimation accuracy. The proposed approach holds potential for tasks requiring precise manipulation in scenarios where visual perception is limited, laying the foundation for closed-loop behavior applications such as assembly and tool use. Please see supplementary videos for real-world demonstration.
An example task that requires tactile extrinsic dexterity. A proper grasp is essential when using an Allen key to apply sufficient torque while fastening a hex bolt. The proposed method utilizes tactile sensing on the robot's finger to localize the grasped object's pose and also regrasp the object in hand by pushing it against the floor - effectively leveraging extrinsic dexterity.
Overview
Our framework - Simultaneous Tactile Estimator-Controller - uses tactile sensing to estimate the pose of the grasped object and compute the motion plan to achieve a desired grasp.
Inputs
Measurements
Vision-based Tactile Sensing (Tactile Image)
Robot's Proprioception (Gripper Pose)
Priors
Goal Configuration (Desired Grasp / Pose)
Environment Prior (Orientation/Height of the Environment)
Physics Model (e.g., Coefficient of Friction)
Outputs
Object Pose Estimate
Motion Plan that Achieves the Goal Configuration
Demonstration of the proposed method on the Allen key example. The orange silhouette illustrates the estimated object pose, while the red silhouette depicts the goal grasp. The superimposed grey rectangles visualize the motion plan for the gripper trajectory.
Framework
Graph architecture of the Simultaneous Tactile Estimator-Controller
The framework comprises two main components:
Discrete Pose Estimator:
Computes a probability distribution within a discretized grid of relative gripper/object poses. It uses the Viterbi decoding algorithm to find a filtered sequence of the relative poses given a stream of tactile measurements.
Continuous Pose Estimator-Controller:
It serves a dual purpose.
Takes as input the filtered probability distribution of relative gripper/object poses and outputs a higher resolution pose estimate.
Computes an iteratively updated motion plan in a receding horizon fashion to achieve the goal configuration.
It uses the Incremental Smoothing and Mapping (iSAM) algorithm, which is based on the factor graph model, as the computational backbone. The factor graph allows flexible formulation where estimation and control objectives can be optimized simultaneously as part of one single optimization problem.
Please see the paper for more details on method and results!
Additional Videos