Superhuman Bilateral Surgical Peg Transfer

Using Depth-Sensing and Deep Recurrent Neural Networks

Minho Hwang, Jeffrey Ichnowski, Brijen Thananjeyan, Daniel Seita,

Samuel Paradis, Danyal Fer, Thomas Low, Ken Goldberg

Code: [Link] | 3D models: [Link]

Abstract

Peg transfer is a well-known task from the Fundamentals of Laparoscopic Surgery (FLS). While human surgeons teleoperate robots to perform this task with great dexterity, it remains challenging to automate.

We present an approach that combines recent innovations in depth sensing, deep learning with Peiper's method for computing inverse kinematics and time-minimized joint motion. We use the da Vinci Research Kit (dVRK) surgical robot with a Zivid depth sensor to automate three variants of the peg-transfer task: unilateral, bilateral without handovers, and bilateral with handovers. We use 3D-printed fiducial markers with depth sensing and a deep recurrent neural network to calibrate the dVRK to less than 1 mm.

We report experimental results for 3384 block transfer trials with an expert surgical resident and 9 volunteers using the dVRK robot. Results suggest that the robot can achieve accuracy on par with an experienced surgical resident and is significantly faster and more consistent than the surgical resident and volunteers. Additionally, the robot exhibits the lowest variance in its distance traveled and trial completion time without any collision.

To our knowledge, this is the first automated surgical system to achieve "superhuman" performance in speed and consistency.

Task Definition

To perform the task, the robot moves each of the 6 blocks from the 6 left pegs to the 6 right pegs before moving them back.

We define the following variants of the peg-transfer task:

  • Unilateral (Single Arm): This is the variant considered in most prior works

  • Parallel bilateral (Two Arms): Two transfers can be performed simultaneously. We propose ways to speed up the robot on this task.

  • Bilateral handover (Two Arms): Each transfer consists of a pick followed by a handover followed by a place. This is the standard surgeon training task in the FLS curriculum.

In this work, we consider the bilateral handover variant of the peg-transfer task for the first time in an automation setting. These two settings require precise coordination between two arms to avoid collisions and to efficiently execute handovers if needed.

Robot Calibration

Estimating Surgical Tool Configuration

We use an RGBD camera to track fiducials of colored spheres attached to the end effector. The two spheres on the shaft allows us to decouple the first three joints q1, q2, q3 from the last three q4, q5, q6. We design and place the four spheres on the jaw where they cannot overlap in the camera view within the working range of joints.

Given the point clouds provided by the camera, we cluster the six groups of point sets by masking the color and size of each sphere. We then calculate each sphere's 3D position by formulating a least-squares regression. We calculate six joint angles of the robot based on the dVRK kinematics.

The measurement accuracy is 0.32 mm in sphere detection and less than 0.01 radians in joint angle estimation.

Data Collection and Training

We collect a training dataset D consisting of random, smooth motions of the arm. This long trajectory is first collected by teleoperating the robot in random, continuous motions, densely recording the waypoints, and then replaying them with the fiducials attached. This enables us to collect ground truth information for trajectories that are not specific to peg transfer.

During the process, we collect the physical configuration q_phy estimated from the fiducials and commanded joint angles q_cmd to compile a dataset D. The dataset consists of 2809 data points, and takes 24 minutes to collect.

To estimate q_phy without the fiducials attached, we train a function approximator f using a Long Short-Term Memory (LSTM) with 256 hidden units followed by two more hidden layers of 256 units, which was the best model identified in prior work.

Perception and Grasping Planning

To enable perception from a camera in a greater variety of poses, we use a point set registration algorithm, Iterative Closest Point (ICP), which is simple and computationally efficient.

We obtain the 12 peg positions by cropping a height range with respect to the base frame of the pegboard. Given the desired peg position and the known height of the block, we cluster a point set on the block to be transferred and get a transformation from the camera to the block using ICP.

After detecting the blocks, we find a best grasp among a total of the six potential poses per block. We subdivide each block edge into two potential grasps and pre-calculate the six grasps for the 3D block model. We transform these grasping poses with respect to the block's frame and find the farthest point among the poses that allows joint limits.

Non-Linear Trajectory Optimization

Closed-form Inverse Kinematics Solution

When computing a trajectory for the robot, we convert end-effector poses (e.g., for the pick location) to robot configuration using an inverse kinematic (IK) calculation. General IK solvers typically treat this problem as an iterative optimization problem and thus may take many iterations to converge to a solution. Due to the kinematic design of the dVRK robot, we observe that a closed-form analytic solution exists that allows for fast IK solutions.

Using Peiper's method, We inversely re-assign the coordinates. We consider the end effector as a base frame. This method can be applied to any 6 DoF surgical manipulator with RCM motion constraint.

We get a computing time of 0.8 ms in average with the proposed IK solution, while 116.3 ms with the numerical optimization method. We used SciPy module in Python for the comparison.

Cubic-backed Time Minimization

We optimize a time-minimized trajectory to improve the speed of task performance. In prior formulations of pick-and-place trajectory optimization, a time-minimized trajectory is found by discretizing the trajectory into a sequence of waypoints and formulating a sequential quadratic program that minimizes the sum-of-squared-acceleration or sum-of-squared distance between the waypoints.

We observe that this prior formulation, while versatile enough for the peg transfer tasks, can be simplified to minimizing a sequence of splines defined by up to four waypoints, and relying on the kinematic design of the robot to avoid collisions. By reducing to four waypoints, we tradeoff some optimality of motion for the reduced computational complexity of the trajectory optimization. In this work, we pursue a fast and computationally efficient method to build a trajectory that can run during the peg transfer task.

Physical Experiment

Superhuman Performance Comparison

We analyze the performance of 9 volunteers, a surgeon with more than 1200 hours of da Vinci experience, and the automated robot, in terms of transfer success rates, tool-tip distance traveled per trial, number of collisions per trials, and completion time per trial.

The leftmost plot shows the mean success rate, with one standard deviation error bars, while the plots to the right utilize a traditional boxplot. On average, all three succeed on the vast majority of transfers, but the robot is more reliable than all humans for the bilateral variant. While distance traveled is comparable for the surgeon and the robot, in terms of collisions the robot is superhuman for all three task variants, and for completion time in the bilateral and handover variants. Additionally, the robot exhibits the lowest variance for all performance measures, suggesting that it also achieves superhuman consistency in its motions.

Position Trajectory Examples

These examples show position trajectories of surgical tool-tip during one completion of trial (12 block transfers). The motion of Volunteers tends to travel long and winding. This is because they have several trials and errors until a successful pick-up or placing. The Surgeon makes a relatively smoother trajectory but has jiggling in motion. The Surgeon often has an overshoot that passes through the target peg and returns to it. The Robot creates a smooth trajectory which is consistent during all trials.

Unilateral (One Arm)

Parallel Bilateral (Two Arms)

Handover

Video Archive

Unilateral (One Arm)

Volunteers

volunteer_unilateral_Raven3.mp4

Surgeon

surgeon_unilateral.mp4

Robot

rob_cal_opt_unilateral2.mp4

Parallel Bilateral (Two Arms)

Volunteers

volunteer_biilateral_Raven6.mp4

Surgeon

surgeon_bilateral.mp4

Robot

rob_cal_opt_bilateral7.mp4

Handover (Two Arms)

Volunteers

volunteer_handover_Raven3.mp4

Surgeon

surgeon_handover1.mp4

Robot

rob_cal_opt_handover4.mp4

Questions?

Contact Minho Hwang(gkgk1215@gmail.com) or Ken Goldberg(goldberg@berkeley.edu)

to get more information on the project