Learning to Localize, Grasp and Handover Unmodified Surgical Needles

Albert Wilcox, Justin Kerr, Brijen Thananjeyan, Jeff Ichnowski, Minho Hwang, Samuel Paradis, Danyal Fer, Ken Goldberg

*Equal contribution

Paper | Code (Coming Soon)

Abstract

ICRA_2022_needles_submitted.mp4

Robotic Surgical Assistants (RSAs) are commonly used to perform minimally invasive surgeries by expert surgeons. However, long procedures filled with tedious and repetitive tasks such as suturing can lead to surgeon fatigue, motivating the automation of suturing. As visual tracking of a thin reflective needle is extremely challenging, prior work has modified the needle with non-reflective contrasting paint. As a step towards automation of a suturing subtask without modifying the needle, we propose HOUSTON: Handoff of Unmodified, Surgical, Tool-Obstructed Needles, a problem and algorithm that uses a learned active sensing policy with a stereo camera to localize and align the needle into a visible and accessible pose for the other arm. To compensate for robot positioning and needle perception errors, the algorithm then executes a high-precision grasping motion that uses multiple cameras. In physical experiments using the da Vinci Research Kit (dVRK), HOUSTON successfully passes unmodified surgical needles with a success rate of 96.7% and is able to perform handover sequentially between the arms 32.4 times on average before failure. On needles unseen in training, HOUSTON achieves a success rate of 75-92.9%. To our knowledge, this work is the first to study handover of unmodified surgical needles.

The Bimanual Regrasping Problem

Bimanual regrasping, the task of handing a needle from one arm of a surgical robot to another, is an important task that must be completed reliably in order to successfully automate multi-throw surgical suturing. However, there are a variety of factors that make this problem difficult:

Varying cable tension in the arms of the surgical robot makes it impossible to perform highly accurate motions, which makes high precision tasks like needle handover difficult to automate
Although accurate 3D pose information is necessary to successfully complete needle handovers, this information is difficult to obtain from active depth sensing because needles are small, thin, and reflective

The HOUSTON Algorithm

The HOUSTON algorithm consists of multiple phases using a mixture of coarse and fine-grained controllers to reliably and accurately hand surgical needles between end effectors on the daVinci Research Kit surgical robot. The main components include:

A perception algorithm using deep convolutional networks to predict segmentation masks on stereo pairs of images, which are used to accurately estimate the needle's 3D pose information
An algorithm for iteratively moving the needle to positions where it will be easier to estimate its pose information
An algorithm for moving the needle to a position where it will be easiest to grasp with the other end effector
A learned policy for completing the high-precision grasping part of the handover

Experiments

The experiments aim to answer the following question: how efficient and reliable is HOUSTON compared to baseline approaches? To answer this question, we compare to a variety of baselines and ablations, described in more detail in the paper.