Create your own image classifier with Angular and Tensorflow.js

If you a JS developer that was always amazed by things Data Scientist doing with Machine learning and Artifical intelligence. Check out this blog and learn how you can create your own image classifiers using only Javascript in less than a few minutes!

AngularInDepth is moving away from Medium. More recent articles are hosted on the new platform inDepth.dev. Thanks for being part of indepth movement!

AI (Artificial intelligence) and ML are definitely the biggest buzzwords of the last few years.

It feels like everyone talks about AI all around. But feelings are not facts, so, I’ve checked google trends to witness the fact that those two terms are kept raising:

Somehow, it always feels for me like this topic reserved for only data scientists, python developers or anyone else or are much smarter than me. So, I’ve checked the related queries at google trends:

No frontend, no javascript, no Angular. Mostly Python.

I have to admit, I always was amazed by the thing people doing with AI and as a front end developer, I always thought to myself why can’t we do AI in JS?

What is AI?

“The theory and development of computer systems able to perform tasks normally requiring human intelligence, such as visual perception, speech recognition, decision-making and translation between Languages” (source)

And in other words: everything that computers do, but, needs the intelligence of a human.

Examples?

Siri, Alexa, Tesla, even Netflix use AI-based concepts to recommend your next tv show.

How does it work?

believe it or not, it’s based in biology.
Our mind is full of Neurons, which getting the inputs from our senses and generating the output through the axon. The important fact is — that, this is what creates the intelligence part of our brain.

Now, let’s try to take this thing, and convert it into a mathematical model.
How can we implement a Neuron using code in the most simple way?
It’s probably a node which getting some inputs and generating them into one output.

In most of the cases, one Neuron wouldn’t be enough for creating the prediction model we want. So we actually converting it into a neuron network which looks like this:

Any input value getting weight on the nodes, and by running some activation function we getting the output on the second edge.
But, let’s say we know that the output we got is not the expected one, o we need to recalculate the weights on the nodes so the model will fit our example. We actually pump up the numbers again for the same example so we will get the excepted result on our next iteration. This process called the training, and that’s how our model is getting improving every-time we adding more data into it.

One of the most efficient ways to create those neuron networks is a technology called TensorFlow.

And almost over a year ago Google announces Tensorflow.js!
At first, I thought Tensorflow.js is sort of a binding to the Tensorflow which is written in C, you have to download and install it on your machine. But, the fact is TensorFlow.js is a complete re-write of Tensorflow, in…JS!

Is JS strong enough to calculate a complex model?

If you asking yourself how the browser can do such mathematical process in a reasonable time? Thanks to a technology called WebGL we can run our JS code using our graphics card memory, which is much stronger and faster than our RAM memory.

we actually can do machine learning in the browser, without any dependency!

Image classifiers

One of the most exciting concepts related to AI is image classifiers.

I was always amazed by the ability of a computer system to understand and classify images as humans do.

Let’s take facebook example — “Someone added a photo you might be in” feature was mind-blowing for me, how the heck they know?!

We just said we can do the same in the browser? Let’s do it!

We’re going to use a pre-trained model, called “mobilenet”. The reason it’s called mobile net is that this model is designed for mobile devices, it loads really fast compared to other prediction models.

We’re gonna create 2 examples:

• Upload an image, and let the browser classify wheat’s in it.
• Wire up our webcam, and let the browser classify what’s we see in the webcam.

Image upload classifier

Step 1: Install TensorFlow & Mobilenet

npm i @tensorflow/tfjs
npm I @tensorflow-models/mobilenet

Step 2: Import mobile net in your component

Step 3: Load the mobilenet model

Let’s load the model OnInit and also add loading indication:

Step 4: Prepare the HTML

Now — let’s add our input file section, along with the loading indication to the template:

Step 5: Implement the fileChange() function

I’m adding the imgSrc class property which I’ll get from the FileReader, to show the image uploaded preview.

Step 6: Classify the uploaded image

I’ve got the imgSrc, now I can run the model.classify() method and get the model predictions:

Now, let’s also update the template to show the predictions:

And the result:

Everything works! and we’ve just created our first image classifier with only JavaScript!

Webcam classifier

Mobilenet classification can perform not only on static images, we actually can classify and predict live video stream!

Let’s add an HTML5video tag to our template:

Now, In order to wire our webcam, we have to set the stream. Let’s first get a hold of the video element with @ViewChild:

And implement the AfterViewInit life cycle:

Few things to notice here:

1. Since we’re changing DOM elements we’ve better use the AfterViewInit life cycle.
2. We’re checking if the user got the navigator.mediaDevices.getUserMedia to make sure we support all browsers.
3. We got a promise out of the getUserMedia function which contains the stream, and we set the video nativeElement.srcObject to this stream, and with that, we can see ourselves:

The last thing - Let’s run the classify() method every 3 seconds on the webcam stream:

And the result:

I’ve checked a remote control, a wallet, and a cell phone. You can see that the classifier actually succeed to classify those items!

It’s not 100% accurate, but still very impressive!

Let’s summarise what we just learned:

• You don’t have to be a data scientist to do AI any more!
• TensorFlow.js is an independent package, you can run it in the browser with a matter of a 1 simple import
• Is the future of FE developers is taking part in building AI-based prediction models? I’ll put my money on that :)

Live demo

I’ve created a stackblitz repo with the examples we’ve just created:

StackBlitz

Visit my website for more content.
Contact me on Twitter.

Thanks for reading!

More references:

1. Me talking about ML in JS on DevDays Europe 2019:

2. Loves packman? Let’s play packman…In a different way…
One of the coolest projects I saw is a packman game you can play with training a model to move into 4 different directions depending on the images you’ll provide for each direction. (Link)

3. Asim Hussain hectic talk from AngularConnect 2019 about How to AI in JS: