Which cat breed is this? A more complex test case for the cognitive computing
Earlier this week I’ve written about a simple machine learning experiment that tries to determine if a given image is a taco or not.
Ever wondered how to get your hands on machine learning (ML)? Well, now you can through machine learning services…medium.com
Using the same cloud based cognitive machine learning technology, Custom Vision, I’d like to tackle a more complicated scenario.
Can a machine learning algorithm determine a cat breed?
Training data sets
As per usual, I need to feed sets of images for a machine learning algorithm to understand what it should learn.
First, I found a list of cat breeds with their names and photos for me to use as a basis to look for images online.
Then, I went to Flickr and searched the name of each breed and used Chrome Extension plugin to download images from the first page.
The reason why I used Flickr over Google Images was to download images in high resolutions.
It was also easier to target user uploaded images on Flickr that are more practical to test against than unrealistically edited images of cats.
After spending an hour or so to download photos of cats, I had to curate them.
Obviously, I had to remove all images that are not a cat so they don’t affect the training sets.
Then, I went to weed out images of the same cat in the same scene, literally just a burst shot of the same cat.
That was to avoid over-training the algorithm where it looks for an exact match of the photo rather than being more predictive about it.
Remember, the purpose of machine learning is for the algorithm to identify new information that is never seen before.
The last part I had to make sure was to collect at least 5 images of a particular breed since that’s how many images you need for each tag you create on Custom Vision.
I’ll admit some biases in data because some common cat breed had over 20 images while some uncommon breed only had just over 6 usable images.
That could skew the result of identifying a cat breed, but at least we are aware of potential bias in the algorithm.
So in the end, I collected just over 900 cat photos for 50 different cat breeds.
Once I finished the data collection, I then upload all images to Custom Vision and tagged each photo with a corresponding breed.
After that, you just click a button to train the algorithm and it’s ready to run a prediction for you in seconds.
Testing data sets
This is where fun begins, let’s first look at the Performance of the algorithm.
Based on 900 images and 50 tags of breeds, the performance of the algorithm is expected to work so-so.
I expected a low level of performance given that it is more than identifying a cat apart from other objects.
The nuance of detecting differences in cat breed would be difficult for humans that are not trained to spot differences either.
So with a grain of salt, I decided to test the algorithm against images that the algorithm has never seen before.
First up is Sola, my Scottish Fold cat with straight ears (I know)
First, I tested the algorithm against my cat which I know for sure that she is a pure Scottish Fold breed.
It’s interesting to test my cat against training sets because all Scottish Folds in training sets have a folded ears.
I doubt that had much of an effect, so I carried on to test close up photo of my cat next.
All of a sudden, my cat is being predicted as a high chance of being Cymric.
So I checked with all images tagged as Cymric in my training sets and found an image of a cat that looked exactly like her.
Even though cat breeds have their distinct look, it’s always possible that two cats from different breed look very similar in the facial pattern.
In this case, the match was unfortunately coincidental, and perhaps more variety of Cymric images would have avoided such issue.
Successful test cases
Let’s take a quick look at some successful test cases across different breeds below.
I must say it feels amazing when the algorithm is spot on with random breed images you test against.
The algorithm is not without a flaw, though, so let’s take a look at some tricky cases where it fails.
Failure test cases
Let’s give a spotlight to Siamese cat since the algorithm performed extremely well in the earlier example.
First, how about an image of Siamese from the back?
This is not too much of a surprise, to be honest.
All the training image sets for Siamese have their faces looking toward the camera instead of looking away to the side.
How about a close up face of the Siamese?
This result seems similar to the earlier test of my cat’s close-up face.
Perhaps the algorithm does work better when it see more than just a face of a cat to determine the breed.
Honestly, I thought it’d do little better than 11.1% given that Siamese training sets had similar images.
Nevertheless, let’s move on to test non-cat images.
Dogs, Bunnies, Guinea Pigs
You might have realized that there is a ‘cat’ tag for each result.
With Custom Vision, you are required to tag each image with at least two tags.
So let’s put that also to a test with other furry animal photos.
Apparently, the algorithm has hard time distinguishing what’s cat and what’s not if they look similar enough.
Quite possibly, the trained algorithm is fuzzy matching what it sees based on colors, patterns, and overall shape.
Given a just a rough training sets, the trained algorithm performed well under certain conditions identifying cat breeds.
However, it did not do well with what it’s not trained to do such as telling apart cats and non-cat animals.
So before anyone invests their time in training any machine algorithms, it seems to be wise to define the scope of what it should be able to handle.
Do you own cats and curious to see how the algorithm performs with their photos?
Feel free to send me an image URL in the comment and I’ll run the test for you!