Learn and Play with TensorFlow.js [Part 4: Application Example]

Published in

Artificial Intelligence Laboratory — Telkom University

8 min readMay 7, 2019

We now have learned how to build web applications that implement Machine Learning capabilities using TensorFlow.js from the previous chapters. We have tried it all, starting from designing and building the model, then we train it, test it, and even save and load the trained model.

Not bad for a beginner.

But we can see, especially in chapter three, that training Machine Learning sometimes can be heavy and time consuming. If we remember from the introduction, the main purpose of building the TensorFlow.js library is actually not for the training process, but rather for the process of testing and implementing models that have been trained.

The process of training the model itself is generally carried out on highly capable computers with stacks of Memory and GPUs, with more suitable and efficient programming languages such as Python. After the model is trained, then we use TensorFlow.js that provides an easy facility for us to be able to use the model directly within a web application interface. This will make the Learning Machine application directly accessible to any users through browsers that can be opened in computers or mobile devices.

Using Pre-trained Model

There are several easy ways to use the pre-trained model in TensorFlow.js. First, you can train yourself a Machine Learning model using TensorFlow in Python language. Then you can convert the trained models and save them as .JSON and weights files that are recognized by TensorFlow.js. Then to use the model, you can just upload the model files into a cloud storage server, and your model is ready for use.

Another way is to use models that have been provided and shared by TensorFlow team as an example of an interesting applications. Those models are free to use and we can directly use them. Furthermore, there are also many TensorFlow.js users and developers out there who have trained their own models and share it so anyone can use it.

Next, let’s look at some examples of popular and awesome web applications that were built using TensorFlow.js. You can get all the examples here. The models used here includes several well-known state of the art models in Machine Learning. All of these models have been trained using the popular Machine Learning Learning Framework such as TensorFlow and Keras.

Let’s try it

Hand sign Digit Recognition

Our first example is a Simple Hand sign Recognition application that run directly through a webcam to recognize numbers. The model used is a model that has been trained by the Telkom University Artificial Intelligence Laboratory in Python using Keras library with TensorFlow as its backend. The model then converted to TensorFlow.js format using tfjs-converter. We use MobileNet architecture which is known to be lightweight but still has a high performance.

To alleviate the case, the recognition is limited to only recognizing the number signals from 0 to 5. Oh, and we trained the model using dataset that we obtained after we enrolled in deeplearning.ai course by Andrew Ng,

so, kudos to Andrew Ng and his deeplearning.ai

If you look at the model stored in here, you will see that the converted model is a bit different from model that we saved directly using TensorFlow.js. The converted model contains a single .JSON file and three parts of weights file called SHARD. The number of shards created during conversion depends on the size of the model being converted.

As explained earlier, to use the trained model, we need to upload the entire model to a storage server. Then to use it, we just call the link to the json file, and TensorFlow.js will automatically read all other weight files listed in shard.

So you can actually load the model that we’ve uploaded in GitHub through the json file in this link. Or if you want to run the application and the model locally, then you need to install a library that will simulate cloud server storage. For that we can use http-server library in Node.js. To install the library simply open a new command prompt, and enter the command below

\ai_tfjs> npm install -g http-server

After the installation is completed, change the current directory into the saved_model directory, then we can simulate the storage server in that directory by running the following command

\ai_tfjs> cd saved_model
\ai_tfjs\saved_model> http-server -c1 --cors

Now all the contents of the directory will be accessible through local address http://localhost:8080/

Next we just need to turn on this web application by executing the command

\ai_tfjs> parcel part_4\handsign.html

The application view will look like the following image. You just need to insert the link to the json location, whether it’s from your local storage or the one in our GitHub, then load the model. You can browse the example images, or use your webcam and try it real time.

Note that the model is trained using image with white background, so it might not perform too well in a cluttered image

Painting with Neural Style Transfer

Next, let’s look at the implementation of Machine Learning techniques to produce results other than regression and classification, namely the Generation technique. This technique really get its hype in the last couple years.

Here, we use two models called the Style Model and Transformer Model. Both models are trained using technique known as Neural Style Transfers so that they can paint (generate) a new painting.

Well, long story short, the way this application works is by giving two images called content image and style image. Style Transfer will try to re-paint the objects in the content image, but with the “painting style” according to the given style image. Painting style can be interpreted as the style of choosing colors, objects, painting strokes, shades, and so on.

The model used in this example comes from a model trained by a contributor named Reiichiro Nakano. We then converted the model so that it could be used on current TensorFlow.js version.

At first the style image will be fed into the Style Model to quantify the characteristic features or style of the painting. Then these style feature results, together with the content image will be fed into the Transformer Model to generate the new image. The Transformer Model will try to generate a new image that will contain object features similar to input content image, while maintaining the resulting image style is similar to the given input style image.

To run this Style Transfer web application, execute the command below

\ai_tfjs> parcel part_4\styletransfer.html

In the application, we can change style of painting and adjust the detail of the style that will be displayed by resizing both the input image of the content and the image of the style. We can also adjust the transformation details when the model is adding the new style using the Stylization Strength slider

Toxic Sentence Classification

Now we have tried using the model that we have trained ourselves. Next, let’s try the sample model that has been provided officially by TensorFlow.js. Some of these official examples from TensorFlow.js have been published via the TensorFlow GitHub account on the tfjs-models and tfjs-example repositories. These examples generally already contain complete helper functions that’ll make it easier for us to try and use them.

So far, we have tried cases that mostly use images as input, or a field commonly referred to as Computer Vision. For this third example, let’s try another case which is a case where the input is text. On March 16, 2019, TensorFlow released a model that was able to classify whether the text contained toxic sentences (insults, threats, profanity, insults, etc.) or not. The published model is titled Toxicity Classifier, and the model shared is already in TensorFlow.js format.

So let’s try it

The model is shared through npmjs package, and to use it we can simply install the package using npm install command.

\ai_tfjs> npm install @tensorflow-models/toxicity

After the package has been downloaded, execute command below to run web application

\ai_tfjs> parcel part_4\toxicity.html

The application view when we run it will look like the image below

Pose Estimation using Posenet

Another interesting example of the model shared by TensorFlow.js is the implementation of PosNet to detect/estimate the pose of the human body. This model will try to find the joint positions of humans detected from the webcam and describe the ‘stickman’ from of joint points detected in real time. This model is very useful to detect and recognize human poses and actions, and the implementation can be very wide.

Just like before, to use the PoseNet model, we only need to download the library with syntax

\ai_tfjs> npm install @tensorflow-models/posenet

After the library has been downloaded, run the syntax to try the web application provided

\ai_tfjs> parcel part_4\posenet.html

The appearance of the posenet.html application will look like the image below

Object Detection using Tiny YOLOv3

With the culture of sharing that is now highly promoted in the Computer Science and Machine Learning community, the researchers and application developers become very generous and easy to share the results of their research and trained models. One interesting model that is widely used is Object Detection using a technique called YOLO Detection.

YOLO is an acronym for object detection techniques You Only Look Once. This technique was developed by a researcher named Joseph Redmon. He continued to develop the YOLO model, which first called darknet from version 1, to the latest version 3. And during that time Joseph always shared the models he trained. Therefore many other contributors use and participate in developing it.

Now the model we will use in this example is the model that has been converted into the TensorFlow.js model by a contributors named shaqian. Based on that, we then further define the model and functionality to use one of the YOLO models into a package that we then published to npmjs package site. To use this functionality, we only need to download the package using the following npm install command

\ai_tfjs> npm install tfjs-tiny-yolov3

The model we used here is called Tiny YOLO-v3. The latest version of the YOLO model that is trained using a small, lightweight architecture. After the package has been downloaded, run the command below to try the YOLO Detection web application