Applying Depth-Sensing to Automated Surgical Manipulation with a da Vinci Robot

Minho Hwang*, Daniel Seita*, Brijen Thananjeyan, Jeff Ichnowski, Samuel Paradis, Danyal Fer, Thomas Low, Ken Goldberg.

For questions about the paper, email Minho ( and Daniel ( with details on what you want to do.

Here are relevant resources for the project.

  • [Paper (arXiv) link] If your work is similar to ours, we encourage you to use the BibTeX below.
  • [GitHub code link] The code is not currently set up to be documented for outside users. If you have questions on how to use the code, email Minho and Daniel with what you are hoping to do, and we will do our best to help you out.
  • Here is a (view-only) spreadsheet showing statistics and relevant information from Danyal Fer's attempts at the peg transfer task. This also contains statistics from his hand-over trials, though this is not in the current paper.
  • Here is a (view-only) spreadsheet for the dVRK's automated runs.

Here is the BibTeX which you can copy and paste easily:

    author = {Minho Hwang and Daniel Seita and Brijen Thananjeyan and Jeffrey Ichnowski and Samuel Paradis and Danyal Fer and Thomas Low and Ken Goldberg},
    title = {{Applying Depth-Sensing to Automated Surgical Manipulation with a da Vinci Robot}},
    booktitle = {International Symposium on Medical Robotics (ISMR)},
    Year = {2020}

Here are some videos. For reference, here is the video corresponding to work from Rosen and Ma. All of the videos below are in real-time with no speed-ups.

The automated dVRK performing some episodes. NOTE: be warned, the videos have flashing lights in them due to the overhead camera we used. In the future we will attempt to reduce the number of times the camera flashes. We apologize for that.

Single-arm example. No failures.


Single-arm example. This has a 10/12 success rate, due to two failures on the return set. Note that one of the placing failures was inserted reasonably well but could not touch the bottom of the workspace due to an existing block that was under it. We count this as a place failure as we are strict in our criteria.


Bilateral-arm example. 11/12, one placing failure on the return set.


Danyal Fer's episodes.

Single-arm example. No failures but he was somehow able to get 2 blocks with one attempt. A nice job defending the human race. :-) . There were a few subjective / ambiguous, *potential* pick failures but we didn't count them here, that would be ultra harsh even by our standards.


Bilateral-arm example, two place failures, one grasp failure. (The grasp failure was on the first set and a bit harsh since he quickly got it the second time, but we are being uniformly strict in our evaluation criteria.)


For reference, here are some of Dr. Fer's episodes with the "usual" peg transfer task with hand-overs.