Recent advances in depth-sensing have significantly increased accuracy, resolution, and frame rate, as shown in the 1920x1200 resolution and 13 frames per second Zivid RGBD camera. In this study, we explore the potential of depth sensing for efficient and reliable automation of surgical subtasks. We consider a monochrome (all red) version of the peg transfer task from the Fundamentals of Laparoscopic Surgery training suite implemented with the da Vinci Research Kit (dVRK). We use calibration techniques that allow the imprecise, cable-driven da Vinci to reduce error from 4-5 mm to 1-2 mm in the task space. We report experimental results for a handover-free version of the peg transfer task, performing 20 and 5 physical episodes with single- and bilateral-arm setups, respectively. Results over 236 and 49 total block transfer attempts for the single- and bilateral-arm peg transfer cases suggest that reliability can be attained with 86.9 % and 78.0 % for each individual block, with respective block transfer speeds of 10.02 and 5.72 seconds.

Here are relevant resources for the project.

• [GitHub code link] The code is not currently set up to be documented for outside users. If you have questions on how to use the code, email Minho and Daniel with what you are hoping to do, and we will do our best to help you out.

• Here is a (view-only) spreadsheet showing statistics and relevant information from Danyal Fer's attempts at the peg transfer task. This also contains statistics from his hand-over trials, though this is not in the current paper.

• Here is a (view-only) spreadsheet for the dVRK's automated runs.

Here are some videos. For reference, here is the video corresponding to work from Rosen and Ma. All of the videos below are in real-time with no speed-ups.

NOTE: be warned, the videos have flashing lights in them due to the overhead camera we used. In the future we will attempt to reduce the number of times the camera flashes. We apologize for that. We were able to fix this in our follow-up papers here and here which also study the peg transfer task.

In the videos below, our co-author Danyal Fer (a practicing surgical resident at that time) teleoperated the robot.

@inproceedings{minho_pegs_2020,

author = {Minho Hwang and Daniel Seita and Brijen Thananjeyan and Jeffrey Ichnowski and Samuel Paradis and Danyal Fer and Thomas Low and Ken Goldberg},

title = {{Applying Depth-Sensing to Automated Surgical Manipulation with a da Vinci Robot}},

booktitle = {International Symposium on Medical Robotics (ISMR)},

Year = {2020}

}

This research was performed at the AUTOLAB at UC Berkeley in affiliation with the Berkeley AI Research (BAIR) Lab, Berkeley Deep Drive (BDD), the Real-Time Intelligent Secure Execution (RISE) Lab, the CITRIS “People and Robots” (CPAR) Initiative, and with UC Berkeley’s Center for Automation and Learning for Medical Robotics (Cal-MR). The authors were supported in part by donations from SRI International, Siemens, Google, Toyota Research Institute, Honda, Intel, and Intuitive Surgical. The da Vinci Research Kit was supported by the National Science Foundation, via the National Robotics Initiative (NRI), as part of the collaborative research project “Software Framework for Research in SemiAutonomous Teleoperation” between The Johns Hopkins University (IIS 1637789), Worcester Polytechnic Institute (IIS 1637759), and the University of Washington (IIS 1637444). Daniel Seita is supported by a National Physical Science Consortium Fellowship.