Conclusion

Details

Discuss your results. How well did your finished solution meet your design criteria?

We believe that we were able to meet most of our goals for the project. Our robot was able to play chess against a human player, reading the human's move on the board and then making the next best move, including a piece capture. While we did not end up using standard chess pieces, we believe that our circular pieces were the nearest equivalent and didn't detract from the human player's experience, as they don't have to play with pieces with a bunch of different colors or AR Tags. Additionally, the robot was accurate in its pickup and place motions and there was minimized risk of unexpected collisions. 

That being said, there were quite a few difficulties encountered to reach this stage, and some flaws still remain, as discussed below. 

Difficulties

Did you encounter any particular difficulties?

Camera Transform Issues

One of the major challenges we encountered during this project was determining the transform from the robot to the webcam. Initially, our plan was to position the camera to the side of the chessboard and locate the pieces from there. To compute the transform between the robot and the camera, we intended to use an AR tag as a common reference point. However, we faced difficulties in getting the AR tag to be detected by both the wrist camera and the external webcam simultaneously, and in having that information published correctly in our TF tree.

Following a design change where we opted for a top-down camera view, our approach to capturing the transform from the robot’s base to the webcam had to be modified. This change arose because the AR tag could no longer be simultaneously visible to both the webcam and the wrist camera. We addressed this issue, and the detailed procedure for capturing the transform is described in the implementation section.

Upon reviewing all the calculations and logic related to the transformations, we discovered another issue: the webcam had been calibrated incorrectly throughout the process. Specifically, we had been using the wrong intrinsic matrix for the camera. This was accounted for, and a detailed description of how we did this is noted under the introduction. Upon reviewing all the calculations and logic for the transformations, it was noticed that the webcam was calibrated incorrectly the entire time; we were using the wrong camera intrinsic matrix.

Piece Detection

Piece detection and identification ended up being a much bigger challenge compared to what we were expecting. We encountered significant overmatching of templates to gradients in lighting on the board or background and a single piece being identified as multiple different pieces. Dozens of hours of work went into trying different approaches such as feature recognition, NCC based custom algorithms, and more to get this reliable. We managed to reach a point of relative success and reliability by using the method discussed in implementation, but it is still susceptible to major changes in light and significant missalignment of pieces.

Flaws + Improvements

Does your solution have any flaws or hacks? What improvements would you make if you had

additional time?

To address the challenge of not having the camera transform completed in time, we implemented a workaround. Specifically, we manually captured the real-world coordinates of the four corners of the chessboard (A1, A8, H1, H8). Using these reference points, we were able to compute the positions of all tiles on the chessboard relative to the robot’s base. This approach allowed us to bypass the need for a full transformation between the robot base and the webcam.

However, this solution has a significant limitation: it assumes that all chess pieces are perfectly centered on their respective tiles. In practice, a human player may place pieces slightly off-center or even partially outside the tile boundary. Our current approach does not account for such imperfections, as the robot calculates its grasp point as the center of the tile. As a result, the robot might fail to pick up a piece if it is significantly misaligned with the center.