Machine learning image classification with ml5.js
In late April, after several weeks of COVID-19 lockdown in the San Francisco Bay Area, we seemed to hit the peak of people tweeting pictures of their home-baked artisan sourdoughs. It seemed like everyone had taken up gourmet cooking all at once.
A tweet or two from Tilde Thurium referencing the phenomenon prompted me to consider how a program might analyze these or other food photos and predict which recipes will match a person’s preferences. Useful? Eh, maybe. Fun enough for me to learn a bit about machine learning? I think so.
I didn’t know squat about image classification. Initial Googling suggested that we’d need to train a model on thousands and thousands of photos.
I mostly forgot about it after that for a while.
Then I stumbled on ml5.js.
The ml5.js tutorial videos are goofy, and I mean that with intense admiration.
ml5.js wraps TensorFlow.js and makes it approachable. For all I know, TensorFlow.js is approachable too, but I’m not sure, because I didn’t stumble across any overtly goofy videos for it.
So, I dove in.
Or at least dipped a few toes in.
I started by creating a web page that would allow my kid to draw something and then ask the computer to tell them what it thinks they drew. Results were mixed but it did pretty well with a drawing of a clock face.
Play around with it at https://trott.github.io/wanda-draws/. The code is available at https://github.com/Trott/wanda-draws. Thanks to offloading all the hard work to ml5.js (and painterro for the drawing interface), I only had to write one 89-line index.html file containing about 60 lines of JavaScript.
To create an image classifier, I did this:
MobileNet is a freely-available image classification model that is designed to be usable on mobile and other low-resource devices.
And here’s the code that classifies the image:
So, OK, great, I can use ml5.js to analyze an image with MobileNet. That was easy. I’m making a Medium post out of that?! You bet!
But if you remember Tilde’s original tweet, we want to train a machine learning model so it can tell different kinds of food from one another. And for that, we’re going to use featureExtractor.
But more on that next time. (Or, if you can’t wait, take a look at https://github.com/Trott/wanda-eats.)
