Implementation

Hardware

We received approval from AUTOLab's PI, Professor Ken Goldberg, to use the AUTOLab lab space along with any lab equipment we would need.

The AutoLab's YuMi IRB 14000 Bimanual Robot, manufactured by ABB.

Each arm is a 7-DOF manipulator with its own metal parallel-jaw grippers.

A Photoneo PhoXi which gives depth and grayscale images.

A flat wooden surface with padding foam on top, covered with a black cloth. The field of view of the PhoXi is demarcated with tape.

A custom built channel with outer dimensions 26.5" x 2.68" x 0.56", made from 2 pieces of wood attached to a 0.5" wide metal channel.

Hardware Customization

The YuMi is originally designed for payloads of roughly 500g, and is not meant to handle large forces on its arms. It has a safety feature that disables the robot when an arm encounters a large enough reactive force, requiring us to restart the program. In initial tests, we found that this force threshold was being crossed every time we tried pushed down to fit the cable into the channel, causing the robot to disable before finishing a trial. To circumvent this, we placed foam blocks underneath the channel to absorb some of the force from the fitting action, as seen in the image below. We found that placing a single piece of foam under the entire length of the channel did not depress enough to prevent the robot from disabling, so we cut the foam into smaller blocks that we spaced out underneath the channel. We used hot glue to secure the foam blocks to the underside of the channel.

Foam blocks attached to the bottom of the channel, spaced out to provide maximum cushion.

Software

Our complete system consists of 6 main steps:

1. Perception: Taking a grayscale and depth image of the workspace.
2. Segmentation: Masking the cable and channel from the grayscale image.
3. Tracing: Creating a point cloud of the cable.
4. Endpoint detection: Identifying the endpoint of the cable.
5. Pick and Place: Picking up the end of the cable and placing it onto the end of the channel.
6. Fitting: Pressing the cable into the channel to ensure a snug fit.

We describe each step in more detail below.

Perception

We use the PhoXi sensor to capture a depth and grayscale image of the workspace. The depth image is represented in code as a numpy array where each entry is the corresponding pixel's distance from the depth sensor, in units of meters. The grayscale image is a numpy array with entries ranging from 0 to 255. The depth and grayscale images have the same dimensions, so every pixel has a corresponding brightness and depth value.

Segmentation

Glare reduction
Due to glare, the reflection from the metal channel often appears brighter than the cable itself in the image. To circumvent this, we identify the brightest point in the image and overlay a black circle around that point, so that it is no longer the brightest point in the image.

Isolation
After the brightest point is covered, the next brightest point is nearly guaranteed to belong to the cable. We execute a flood-fill algorithm from that point to segment the cable.
Flood-fill is a variant of breadth-first search, looking at neighboring pixel values and considering them to be part of the cable if the difference in depth between the current pixel and the neighboring pixel is less than a certain threshold. The search stops when the depth of all neighboring pixels exceeds the threshold or when all neighboring pixels have been visited.
After the cable is segmented, we return to the original brightest point on the image, which was on the channel, and execute flood-fill from there to segment the channel.

The raw grayscale image. The brightest point on this image is actually glare from the channel, not the cable itself.

A black circle overlaid over the brighest point. The new brightest point is now part of the cable.

The output of the flood-fill algorithm, segmentations of the cable and the channel.

Tracing

To remove noise picked up by the flood-fill algorithm, we use color thresholding on the grayscale image to eliminate incorrectly identified points in the image. We use this thresholded grayscale image as a mask on the depth image, extracting the points corresponding to the cable. However, this sometimes removes points on the cable itself, so we use a Gaussian blur on the depth image to fill in lost details and make the cable stand out even more from the background.

The thresholded grayscale image, showing some points on the cable having been removed.

The blurred and masked depth image, showing a clean cable segmentation.

Endpoint detection

We detect cable endpoints by looking for cable (yellow) pixels that are surrounded by non-cable (purple) pixels on three sides. With the large image we get from segmentation, many points on the cable are classified as endpoints, especially near the curves. To circumvent this issue, we compress the image by a factor of 30 to pixelate the image. This makes endpoints much easier to detect using our 3-neighbor method, and we then take the corresponding pixel value in the original image as the endpoint of the channel.

The compressed depth image, allowing endpoints to easily be detected with the 3-neighbor method.

The identified endpoint of the cable, masked onto the original uncompressed depth image.

Pick and Place

Once the endpoint of the cable and channel are identified, we plan a trajectory to pick and place the cable onto the channel. We use three libraries, tracikpy to calculate inverse kinematics, yumiplanning to plan a trajectory for the YuMi, and yumirws to execute the trajectory, which are all part of the core AUTOLab library.

We made several modifications around AUTOLab's version of tracikpy and yumiplanning:

When an inverse kinematics(IK) solution trajectory outputted resulted in large jumps in the joint angles, we discarded the solution and repeatedly queried the solver to generate a new trajectory without jumps.
To prevent the robot from executing an inefficient trajectory, we set waypoints along the path, including a "home" position for each arm that it would return to after a set of motions, and a point vertically above each place and press point to ensure the arm doesn't collide with the channel, moving the channel and disabling itself from hitting force limits.

The pick and place trajectory proceeds as follows:

Set the cable end point as the pick point.
Set the channel endpoint as the place point.
Move the grippers to the pick point.
Close the grippers to pick up the cable.
Move the cable end up.
Move to Z-location above the place point.
Drop the cable end onto the place point.

The identified pick and place points.

Fitting

We then fit the cable into the channel by pressing down along the length of the channel. We identify the second endpoint of the channel using the same flood-fill from before, and compute the slope between the two endpoints to determine the orientation of the channel. We sample n points along this computed line and press down on these points with the grippers closed to fit the cable into the channel. We experimentally determined n = 31 to be reasonable for this channel, but this can easily be changed for other channels.

Pushing on the sampled points along the length of the channel.

Page updated

Report abuse