Untangling Dense Knots

by Learning Task-Relevant Keypoints


Jennifer Grannen*, Priya Sundaresan*, Brijen Thananjeyan, Jeffrey Ichnowski, Ashwin Balakrishna, Minho Hwang, Vainavi Viswanath, Michael Laskey, Joseph E. Gonzalez, Ken Goldberg

*equal contribution

Paper + Appendix: [Link] | Video [Link] | Simulation Code [Link] | Training Code [Link]

Abstract

Keywords: Deformable Manipulation, Computer Vision

Untangling ropes, wires, and cables is a challenging task for robots due to the high-dimensional configuration space, visual homogeneity, self-occlusions, and complex dynamics. We consider dense (tight) knots that lack space between self-intersections and present an iterative approach that uses learned geometric structure in configurations. We instantiate this into an algorithm, HULK: Hierarchical Untangling from Learned Keypoints, which combines learning-based perception with a geometric planner into a policy that guides a bilateral robot to untangle knots. To evaluate the policy, we perform experiments both in a novel simulation environment modelling cables with varied knot types and textures and in a physical system using the da Vinci surgical robot. We find that HULK is able to untangle cables with dense figure-eight and overhand knots and generalize to varied textures and appearances. We compare two variants of HULK to three baselines and observe that HULK achieves 43.3% higher success rates on a physical system compared to the next best baseline. HULK successfully untangles a cable from a dense initial configuration containing up to two overhand and figure-eight knots in 97.9% of 378 simulation experiments with an average of 12.1 actions per trial. In physical experiments, HULK achieves 61.7% untangling success, averaging 8.48 actions per trial.

CoRL 2020 Video Submission

Untangling Algorithmic Supervisor (BRUCE)

We formulate the problem of untangling using an undirected graph abstraction to model cables as in prior work. Each crossing or endpoint in the configuration is a vertex (node) and an edge denotes a vertex pair. The graph encodes the sequence of over-crossings (+) and under-crossings (-). Given this representation, we define an algorithmic supervisor for the task: Basic Reduction of Under-Crossing Entanglements (BRUCE). BRUCE employs two primitives — (1) Reidemeister (straightening) moves and (2) Node Deletion (undo) moves and untangles a densely knotted cable by sequentially undoing the right-most under-crossing, breaking ties arbitrarily.

Graphical LDO Representation

Algorithmic Supervisor

Untangling Learned Policy (HULK)

We train a policy — HULK: Hierarchical Untangling from Learned Keypoints for untangling various knotted configurations, using BRUCE as an algorithmic supervisor. HULK takes RGB cable image observations and outputs predicted keypoints for the cable endpoints and self-intersections. In particular, HULK localizes the first under-crossing relative to the right endpoint and predicts a pull point and pin point with which to perform a Node Deletion move. HULK also predicts bounding boxes for knots in the image, in order to determine a stopping criterion for untangling (no boxes detected). HULK can be trained either in simulation or in real from offline datasets consisting of image, annotation pairs.

Given these predictions, we perform untangling using an open-loop geometric policy that grasps at the pull point and pins at the predicted pin point, and pulls in the direction of the vector between the two in order to loosen a knot. This process is repeated until no crossings (no predicted knot bounding boxes) remain, and the predicted endpoints are finally pulled apart to yield a planar state.

Example HULK Predictions

HULK Keypoint Predictions (Real)

HULK Bounding Box Predictions (Real)

HULK Predicted Actions (Simulation)

We visualize some example predictions of loosening (Node Deletion) moves in simulation based on HULK's keypoint detection. The red and blue points denote predicted pull and pin points respectively, and the green arrow shows the predicted direction along which to pull while anchoring the pin point in place.





Example HULK Rollouts

Overhand

Figure 8

Overhand + Figure 8

Overhand + Overhand

Overhand