ReccyChess Part 1: Creating a Custom Chess Piece Object Detector
Imagine you are walking down the streets in New York, and are looking for somewhere to get some fresh air. What better place to go to than the park! You arrive, enjoying the birds, trees, and the bustling environment of the city, when all of a sudden, you see a chess table. Ah… a chess table. Turns out chess is your favorite game! You confidently walk up to the table, thinking that the competition is no match for your superior chess prowess. You ask for the next game, and you start playing against someone, but he is deceptively good. He uses an opening you haven’t studied, pushes his rook, and applies suffocating pressure on you. You fold to the pressure, lose the game, and stumble out of the park in an embarrassing fashion. Two weeks pass and you reflect on your career ending loss. There is only one problem however, you completely forget the sequence of the game…
The solution to this problem which I am proposing is called ReccyChess. My vision for ReccyChess is a mobile application that is able to scan a physical chess board periodically throughout the game using a camera, and generate a digital representation of the game that is easy to store. This can help people watch their own games, save them for later, or run other ML algorithms and analysis on them. This blog will be the first part in a series of blogs which will document my process of creating ReccyChess.
The first thing that I needed to implement to create ReccyChess, which will now be the main focus of the blog, was an object detection model that could accurately identify chess pieces in images. I decided to use the YOLO-V4 Darknet model framework because it is relatively fast to train, it has a fast inference time, it can run object detection on videos(which is going to be useful in the future), and it could achieve high accuracy on test data. Instead of using the YOLO general model to detect chess pieces, I decided I would fine-tune the model with chess datasets to achieve better accuracy. As for my training data, I used a pre-annotated chess dataset from roboflow.com that had image transformations already applied to it. This way, the YOLO model could better understand the features of each of the 12 chess pieces from different lightings and angles, to increase its accuracy. In order to train the model, I decided to use Google Colab as it provides free gpu to help accelerate the training process.
One thing I did to make the chess dataset easier to work with was create a script called data_formatter.py. Some of its features include cleaning up the name of all the files in the test, train, and validate folders. It can also switch up(or even delete) annotations of a certain class in case you don’t want to train the model on a certain class. More information about data_formatter.py can be found in my Github repository. It is completely optional to use if you are trying to create a chess detection model following the steps in this blog.
The next steps were to set up my environment on Google Colab to run the model. Once I formatted the test, train, and validate data with data_formatter.py, I then zipped them up and uploaded them into my Google Drive.
Then, I cloned the Darknet repository into the Colab virtual machine. After this, I downloaded the preset Darknet weights to save time during the training process. Then, I set the configuration files of YOLO Darknet. There were five such files I had to create: train.names, train.data, train.txt and test.txt(generated by the generate_train.py and generate_test.py in the google drive folder- Contributed by TheAIGuy), and the yolov4-chess.cfg file. I have provided all of these files in the yolov4 folder in my google drive.
Train.names specifies all the classes which the YOLO model should be fine tuned on. Train.data tells the yolo model where the training data, validation data, and backup folders(destination folder where it saves the model weights when training). Yolov4-chess.cfg basically optimized the YOLO model to be trained on 12 classes. Generate_train.py and generate_test.py create train.txt and test.txt, which let the YOLO model know the paths of the training and testing data.
After this, I was finally ready to train the model. I trained my model for 1300 epochs, and ran the mAP precision test on my test data to evaluate my model. I ended up getting 87% accuracy, with over 95% accuracy for all classes except for black king(60%) and black bishop(90%). In the future, to improve this, one thing that I can do is analyze the test data where detections of black kings and bishops were more poor, and make those cases more representative in my training data.
mAP Precision Test:
Here are some pictures of inferences the model made:
Finally, I needed to optimize the inference time of the model. Luckily for me, there is a technology from Intel called OpenVINO Integration with TensorFlow(OVTF), which accelerates inference time of TensorFlow models, such as YOLO Darknet, on Intel hardware(Google Colab vm). I ran inferences on 25 test pictures of chess pieces with and without OVTF set as the backend of the vm’s cpu, and here are the results of the inference times.
As we can see, there is around a 3–4% boost in inference time while using OVTF on average, and in some places it’s even more. This small difference in inference time could really help up when running inferences on videos of chess pieces in future versions of ReccyChess.
What’s Next:
My next step into developing ReccyChess will be to make a model or create some software that can detect the position of the pieces as well, so I can fully reconstruct a chess board given an image. This will come out soon, so stay tuned! I also want to increase the accuracy of my model by perhaps creating my own training, test, and validation datasets, and utilizing multiple different types of chess pieces to make the model work on multiple different chess sets.
Thank you for making it to the last part of the blog! I definitely have learned a lot in the process of creating this custom chess detection model, such as how to create python scripts that interact with the file system, the basic process of fine-tuning a model, Google Colab, and many more things. All in all, I still have a long way to go into perfecting this model and ultimately creating ReccyChess, but I look forward to the challenge!
Github: https://github.com/rahulavasarala/YOLOV4ChessDetection
Colab Notebook:
https://colab.research.google.com/drive/1y1DXCSF1gBsbbKcDqJjf---TgzeCLep0?usp=sharing
Roboflow.com Chess Dataset: https://public.roboflow.com/object-detection/chess-full
Check out Roboflow: https://roboflow.com/
OpenVINO TensorFlow Repo:
https://github.com/openvinotoolkit/openvino_tensorflow
YOLOV4 Darknet Repo: https://github.com/AlexeyAB/darknet
Thanks for reading!