M2M Day 367: This is a bit problematic
This post is part of Month to Master, a 12-month accelerated learning project. For October, my goal is to defeat world champion Magnus Carlsen at a game of chess.
I might have a problem…
Yesterday, I excitingly found an easy way to get my chess code running on faster computers in the cloud. The hope was that, if I can process a lot more data and then train my model on this larger dataset, for more iterations, I should be able to greatly improve the performance of my chess algorithm.
As of a few days ago, I could only get my algorithm to correctly label a chess position as “good” or “bad” 70% of the time. I figured I could get this up to around 98–99% just by scaling up my operation.
However, after running the beefier version of my program overnight, I woke up to find that the algorithm is still stuck at the 70% accuracy level.
Not only that, but after it reached this point, the model continued to slowly improve its accuracy on the training data, but started to perform worse on the testing data.
In other words, after my algorithm reached the 70% mark, any attempt it made to improve actually caused it to get worse…
For example, after 6,600 iterations, the test accuracy was 68.1%:
After 9,900 iterations, the test accuracy was down to 67.6%:
So, it seems that my model, as it’s currently constructed, is maxing out at 70%, which isn’t great. In fact, I’d bet the reason it plateaus here is fairly simple: 70% of the data in the dataset is labeled as “good” and the rest is labeled as “bad”.
In other words, my model performs best simply by guessing that every single chess position is “good”, thereby automatically achieving 70% accuracy.
I haven’t confirmed this explanation, but I suspect it’s something like this.
So, now what?
Well, I was really hoping that I could use the most operationally-basic machine learning model, at a large enough scale, to pull this off. But, either the model isn’t sophisticated enough or my labelling scheme for the data is just nonsensical.
In either case, I have some ideas for how to move forward from here, which I’ll discuss tomorrow.
Read the next post. Read the previous post.
