Design & Methods

Task 1: Box

For this task, the recognition process is relatively simple but important. We can identify both the object to be grasped and the target placement area using color detection with the OpenCV library. The object to be grasped is a red sponge block, and the placement area is a green square. We convert the RGB image to the HSV color space and perform color-based segmentation to isolate the desired regions, then extract the corresponding center points.

Task 2: Peg Insertion

In this task, the visual recognition algorithm we used is similar to the previous task, except that the target color is changed from green to blue. However, during actual execution, due to the camera’s perspective, the center point obtained through color segmentation does not necessarily correspond to the 2 true center of the object (the observation angle may cause a shift of the center point in the image). Therefore, when inserting the object into the hole, we utilized the force sensor on the end effector.

We implemented two methods to achieve the same functionality. In the first method, after the robot arm moves the object above the hole, it descends in 1 cm increments. If the object hits the edge of the hole, the resulting reaction force will be significant, which triggers our collision detection mechanism and causes the current command to be popped from the queue. At that point, the robot arm lifts by 0.5 cm and tries eight surrounding positions (by adding or subtracting 0.3 cm to the x and y coordinates). If the height of the end effector drops below 13 cm, it indicates that the object has been successfully inserted into the hole. The gripper then opens to release the object and returns to the starting position. However, a problem with this approach is that asynchronous command execution may lead to delayed feedback or unintended removal of commands that should still be executed. To address this issue, we proposed a second method. We observed that although the camera angle may cause a deviation between the computed and actual target points, the deviation is typically small. Therefore, we designed a modified approach: the overall structure remains the same as the first method, but when the object encounters resistance at the hole’s edge, we partially open the gripper to 60% of its full width, allowing the object to naturally slide into the hole.

Task 3: Coin Pickup

For this task, we used the OpenCV library to perform both color and shape recognition, which allows us to identify the object to be grasped and the target placement area. The object we aim to grasp is a circular coin, and the placement area is a green square. We used a shape-based segmentor to detect the desired coin and a color-based segmentor to locate the target placement area. 

Task 4: Dollar Bill Pickup

For this part, I still used a color-based segmentor. The dollar bill has a brownish-yellow color, and our target area remains a green square. However, a problem arose: from the camera’s perspective, the table also appears yellow. If we rely solely on color segmentation, the table gets included as part of the object, which leads to large-scale contamination in the segmentation results. To address this, we applied a simple trick—by extracting the second largest contour, we were able to isolate and grasp the dollar bill accurately