Conclusions

Did we meet our criteria?

Unfortunately we did not achieve our original goal.

However, we did accomplish many key elements. We were able to use computer vision to detect the table, the pieces, and the AR tag indicating the the reference height and location with respect to Baxter. We were able to consistently pick up the pieces (if it was anywhere in Baxter's reachable space) and move them to the desired final location. While the resulting final configuration was not perfect, i.e. pieces were overlapping or not quite in the right spot, you can typically see that it resembles the inputted desired final configuration.

Challenges Encountered

While working with the MoveIt platform, we ran into many issues regarding the consistency of the motion planner. There would be times when a motion plan could be found fairly easily on an initial run, but be impossible to find during subsequent runs. Further, the solutions granted by MoveIt were not always the same which could often cause problems if the arm of Baxter reached configurations that prevented future pick and place maneuvers.

In terms of computer vision, one daunting issue that affected us throughout the entirety of the project was the sensitivity of our thresholding and contouring algorithms to changes in lighting. The camera on the right wrist was likely damaged because the images were extremely blurry and unusuable, so we were limited to only using the left wrist camera. That proved to be a good solution, but far from perfect; depending on the conditions, the white balance and/or saturation would be blown out of proportion, making it very difficult for the camera to recognize the edges of the table as well as the colors of the different pieces. Under certain circumstances, dark purple would be indistinguishable from the light brown table. Even with a high quality camera like one found on an iPhone 11, which we used to determine initial threshold values, colors can change depending on the lighting, which is a problem since we are using hard coded threshold values. For reference, refer to the before and after image on the main page. Without consistent contouring, our resulting homography would be faulty and lead to errors in the joint angles returned to the MoveIt commander.

One major problem was with Asimov's (the name of the Baxter robot we used) URDF. It switched the cameras -- the camera feed on the right wrist came through the topic /left_hand_camera, and the camera feed from the left wrist came through the topic /right_hand_camera. This was overcome by making our own subscriber and publisher, but it was a major challenge because we needed the AR Tag imaged by the camera on the l left wrist to stay constant with respect to the left arm. This inspired the following meme.

Areas of Improvement

Our current implementation utilizes a four-point homography in order to map the table from the camera feed onto a flat 2D plane. Although four points is generally sufficient since we could just capture the four corners of the table, the resulting 2D image becomes very reliant on perfect corner detection. If we were to perform the homography using more than just the four corners of the table, we would be much more robust to imperfect corner and edge detection.

In our planning phase, we are generating a motion plan for all of our pieces concurrently such that we would only need to collect information at the beginning of each puzzle solve. However, for larger scale puzzles, it could be very likely that pieces could be accidentally shifted from their original positions and thus have a different pose than what was used in the motion plan. It could be very beneficial to have some dynamic method to keep track of the locations of each piece and update the motion plan accordingly in order to be robust to these types of scenarios.

The robot in its current state of development does not have the ability to recognize the angle that the pieces are rotated, and so all of the pieces must be in their final orientation prior to pickup. We can develop a solution within the limitations of the imaging system to find the orientation of the pieces and rotate them to the final configuration after pickup.