Chess Piece Detection Using Faster R-CNN

Colin Acton, Bob Xiaohai Hu, Yifan Liu

Summary

Have we become bored with chess with humans? This project takes the first step in trying something different!

We are building a robotic chess player with a universal robot UR5e, a 2D camera, and a deep-learning neural network at the Mechatronics, Automation, and Control System Lab at the ME department, University of Washington. One crucial step is to detect the pieces on the chessboard and locate each piece on the board. Utilizing computer vision techniques and Faster R-CNN, the algorithms created for this project classify chess pieces and identify their location on a chessboard. The final application saves images throughout to visualize the performance and outputs a 2D image of the chessboard to see the results (see below).

Here we use FAIR’s (Meta AI Research) Detectron2 for target recognition on a Chess Pieces Dataset provided by Roboflow. Then the trained model is applied to the user-defined data set to observe the output.

Code for this project on Github

Problem Setup

The Mechatronics, Automation, and Control Systems laboratory in the Mechanical Engineering department has a vision-based robotic chess player capable of detecting the state of a chess board, finding the ideal move using Stockfish, and executing the move using a UR5e robotic arm.

The goal of this project is to improve upon the Resnet-34 model currently used for state detection. To do so, we use a convolutional neural network to train a model capable of categorize all pieces on a chess board.

Techniques and Setup

Dataset

A public domain, pre-labeled set of images (Chess Piece Dataset) hosted by Roboflow was used to train and test the model. The set consists of 292 unique images with 2894 annotations corresponding to regions containing pieces. The set is augmented with random brightness adjustments and other image effects to yield a total of 693 JPG images for training and testing. 606 images are used for training, 58 for validation, and 29 for testing.

The bounding boxes of all pieces are annotated as follows: white-king, white-queen, white-bishop, white-knight, white-rook, white-pawn, black-king, black-queen, black-bishop, black-knight, black-rook, black-pawn. 12 kinds of chess piece, 1 class for none piece, another empty class added due to Detectron2's requirement, so in total 14 categories were used for training.

|:-----------:|:-------------|:------------:|:-------------|:------------:|:-------------|

| pieces | 0 | none | 3 | black-bishop | 291 |

| black-king | 303 | black-knight | 423 | black-pawn | 1488 |

| black-queen | 186 | black-rook | 441 | white-bishop | 375 |

| white-king | 315 | white-knight | 420 | white-pawn | 1422 |

| white-queen | 243 | white-rook | 414 | | |

| total | 6324 | | | | |

Framework

We use Detectron2, an object detection framework created by Facebook AI research and implemented in PyTorch. It can provide high-performance and high-quality codebase for target detection and image segmentation. Its design is very flexible to support the rapid implementation and evaluation of new research. Detectron2 contains multiple models and collaborate well with Google Colab. These features make it better suited to our project.

Configurations

Models Training

In order to use GPU, we created and trained CNN model in the cloud, which significantly reduced the training time.

First, we import our dataset and register them for convenience. To verify the correctness of our data sets, randomly select samples and annotate them:

Next, we select models from detectron2's model zoo and fine-tune them on the dataset. We have selected several different models and parameters, and it turned out that not all of them are meaningful. The final model used will be discussed in the next section. Here, taking rpn_R_50_FPN_1x model as an example, we set parameter configuration for training.

After completing the training, we output its training curve with tensorboard and test our model on the verification set. Through the test results, we can decide whether to change the parameters and retrain. At the same time, some pictures are generated to make the test results more intuitive, which provides help for further correction of the code.

Feel free to use our code to train your model!

Google Colaboratory

R50-FPN, 3x, 4000 iterations

Google Colaboratory

X101-FPN, 3x, 2000 iterations

Google Colaboratory

RPN_R50_FPN,1x,2000 iterations

Google Colaboratory

R50-DC5, 3x, 2000 Iterations

Application to our system & Testing

Custom Testing Dataset

A chess set similar to the one used in the training data was procured, and 11 images were captured to test the accuracy of our trained models in a lab environment.

Almost all pieces are present in these images, presenting more of a challenge for the model. Densely packed pieces can lead to obscured features.

The images were taken at a slightly different pitch to the board than in the training data with flatter, slightly orange-tinged lighting conditions.

Once our convolutional neural networks have completed training, we export the final weights as .pth files and load them into another Colab notebook for testing on our custom dataset and comparing models.

In the testing notebook, we first set up our environment by installing packages and dependencies. Next, we clone the weight files from our GitHub repository (shown as follows) and download the Roboflow dataset.

With all necessary files loaded into the Colab session, we can create a predictor using any of the configurations simply by running one of the cells corresponding to a pre-trained model, the threshold for the prediction can be adjusted prior to running the model on our test data. (The loading code show as follow)

Google Colaboratory

↑ Click on the image link to view and run our models yourself

Results

Comparison of model results

We have conducted iterative training for four different models for many times, as shown in the figure, which shows the maximum accuracy and minimum loss of each type of model after adjusting the parameters and the number of iterations. The accuracy of some models is not shown because they are too low to be used.

From the performance of the model in the test set, the best is rpn-r50-fpn, but when we apply the trained model to the user-defined test data set, the fastest rcnn-r50-fpn has the highest accuracy. Therefore, for this project, we finally decided to adopt fast rcnn-r50-fpn as our final model.

R50-FPN, 3x

Results

500 iterations

4000 iterations

Discussion

Challenges

When we train the model in the early stage, it takes a long time to iterate, and the number of iterations required by the model is very high. When using the free colab account training model, we often reach its usage limit. Therefore, we should be extra careful when increasing the number of iterations of the model for training. Therefore, we choose RPN network, whose sliding window is on the convolution layer feature map, and the dimension is many times lower than that of the original image. This uses less time while achieving higher accuracy. RPN network and Faster R-CNN share the convolution layer to reduce the amount of calculation and save memory.

Our strength

For Faster R-CNN, ROI acts on the last layer. Because of the unique characteristic pyramid network structure of FPN, when the convolution pool is turned to the last layer, the semantic information of small targets can still be retained. This is very important for our chess recognition problem. It ensures that the model has high detection speed and high accuracy.

In addition, when testing with our own test set, we can adjust the direction of the picture and the light source conditions. In fact, when they are applied to the mechanical arm, it will have a key impact on the accuracy of the mechanical arm. Because in order to ensure the practicability of the system, we should ensure that our model can still have stable accuracy when the image input environment changes.

The change of light conditions and image direction has little effect on the accuracy(the first on the left is the original image)

Next Steps

The goal of this project was to train a model for use in our Robotic Chess Player. The models we trained in this project performed well, however, we were somewhat limited by the Google Colab usage limits. Before implementing them in our ROS package, we would like to run some of our neural network configurations for significantly more iterations to see just how accurate we can get with our predictions.

Additionally it would be best for our application to create our own custom training dataset. We used the Roboflow set because taking and labeling our own images wasn't feasible for the time-frame of this project.

Once we have an ideal model, we will replace the Resnet-34 model within our State Detection Service node in the Robotic Chess Player ROS package.

References

[1] Detectron2 Faster R-CNN Information: https://paperswithcode.com/lib/detectron2/faster-r-cnn

[2] Roboflow Chess Pieces Dataset: https://public.roboflow.com/object-detection/chess-full

[3] R-CNN: https://paperswithcode.com/method/r-cnn

[4] Faster R-CNN: Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks

[5] Chess player robot repository: https://github.com/macs-lab/robotic_chess_player

[6] Chess Pieces Recognition using CNN: https://towardsdatascience.com/board-game-image-recognition-using-neural-networks-116fc876dafa

[7] Detectron2 official Github repository: https://github.com/facebookresearch/detectron2

From left to right Colin Acton, Bob Xiaohai Hu, Yifan Liu

Email: {actonc, huxh, yifanl34}@uw.edu

Page updated

Report abuse