Getting Started with ML5.js — Tutorial Part I: Image Classifier

_________________________________________________________________

PSA: It is a key value for us at AIxDESIGN to open-source our work and research. The forced paywalls here have led us to stop using Medium so while you can still read the article below, future writings & resources will be published on other platforms. Learn more at aixdesign.co or come hang with us on any of our other channels. Hope to see you there 👋

Machine Learning/ML is changing the world, and it has gradually evolved into a new medium and design material for artists, designers, and creatives; however, algorithms and programming can become a great entry barrier for many to understand and explore the capability that ML offers.

ML5.js exists precisely for this purpose. With their entry-level interface, ML5.js made ML more approachable and easy to use, specifically with designers and artists in mind.

Event poster for Erik’s ML5.js workshops

In November and December 2021, we hosted a series of ML5.js workshops with Erik Katerborg — a creative technologist, lecturer teaching Creative Media and Game Technologies at the Rotterdam University for Applied Sciences, and active AIxDesign community member.

To spread the word and invite more people to make their first steps into coding, we created this article as a written version of the workshop for anyone to follow at any moment in time. If video is more your thing, you can also watch a recording of the workshop on our Youtube channel here.

In Part I, we will look at how and why to use Javascript and ML5.js for your creative Machine Learning projects. Then, you will be able to get the first grasp of some of the ML5.js functions by making your own ML5 Image Classifier game.

What is ML5.js?

ML5.js is a user-friendly library developed based on Tensorflow.js, and heavily influenced by P5.js. As the mix of the two, ML5.js aims to make Machine Learning more accessible for designers, artists, and other members of the creative community. It provides access to many pre-trained Machine Learning models with a few lines of Javascript code.

ml5js·Friendly Machine Learning For The Web

Why Javascript?

Accessible

Traditionally, Machine Learning is done with Python with code editor or command lines. Implementing a pre-trained model and API in a web editor can make the work easily accessible and usable. In addition, the browser can also give you quick access to data from the camera or microphone, and present the data quickly to the user.

Sharable

Making Machine Learning projects using Javascript and implementing them online can avoid dependency or installation issues. Furthermore, your work can be shared, played, and remixed easily, similar to p5.js.

Private

Building everything in a browser can reduce the chance of sharing your data to a python server, which could often lead to privacy concerns.

Creative

Javascript allows your app to be directly linked to an interactive user interface, art project, game, or any other javascript library.

Project Kalidoface allows you to animate Live2D characters using just your browser webcam

Let’s build your ImageClassifier!

The project that we will develop is using ML5.js’ Image Classifier API to build a simple web interface that can detect and classify a user uploaded image.

Now, let’s walk through the workshop tutorial. If you are in a rush, make sure to check out the finished project/code at the end.

Step 0. Setup

To start, go to the template that Erik has set up and click “Remix your own”

On the newly created Glitch page, you will be able to see three panels:

• A list of HTML/CSS/Javascript files on the left
• The code editor in the middle
• A Preview/Show panel on the right (You can toggle this using the top “Show” drop-down menu)

We will not be covering the detail of basic HTML, CSS, and Javascript but we will cover every step you need for the project.

You can start by changing the project name and deselect “Refresh App on Changes” to reduce the load of the browser. To see any changes, press the “refresh” button in the Show panel.

Step 1. Load Image

In the script.js, you will be able to see all the Javascript needed for this project. There are 4 existing variables and each of them has different properties.

const message = document.querySelector(“#message”)
// Controls the message showing on the screen
const fileButton = document.querySelector(“#file”)
// Provides the access to upload image
const img = document.querySelector(“#img”)
// Contains the uploaded image 
const synth = window.speechSynthesis
// Gives sound output

Below, there are three functions included.

function loadFile(event) 
// It will run after the user select the file and load the user file 
function userImageUploaded() 
// It will run after the image is loaded and change the on-screen message to "Image was loaded!" 
function speak(text)
// It will run and read text audiblly (Optional)

To test step 1, click on the “Choose File” button in the Show panel, select your hamster image, and see your image in the browser.

Step 2. Load ImageClassifer Model from ML5.js

Once you can upload an image in the interface, we will load the ImageClassifer model pre-trained via ML5.js.

First, you will need to link the ML5 file to your project. Copy and paste the following line in your HTML file, right under the Title line.

<script src="https://unpkg.com/ml5@latest/dist/ml5.min.js"></script>

It should look like this

Then, go to https://learn.ml5js.org/#/reference/image-classifier. This page documents how you can implement and use the API and we will only need to look at the “Quick Start” section. Let’s copy the first two sections into our script.js to initialize the Image Classifier method and load the model.

By embedding the code into our project, we can load the Image Classifier model and notify the user once the loading is complete. Loading the model may take an extended amount of time, so let’s add a text notification to notify the user to upload an image when the model is loaded.

In the function modelLoaded(), add

function modelLoaded() {
  console.log('Model Loaded!');
  message.innerHTML = "Please upload an image of a hamster!"
}

In the HTML file, change the on-screen message from “Loading” to

<div id="message">Please wait...loading model...</div>

Now if you refresh the Show panel, you should be able to see the message on-screen change from “Please wait…loading model…” to “Model Loaded!” once the Image Classifier model is loaded.

Step 3. Classify the Image

Now, we have successfully set up the image upload function, embedded the ML5.js Image Classifier model, and adjusted the text prompt and notifications in our project. Next, we will take a look at how to let the model classify the image.

Let’s go back to https://learn.ml5js.org/#/reference/image-classifier and copy the last section of the Quickstart code, and paste it into our userImageUploaded() function. We also want to edit the code a bit and change the message to the result of the classification.

function userImageUploaded() {
     message.innerHTML = “Image was loaded!”

     classifier.classify(img, (err, results) => {
         console.log(results);
         message.innerHTML = results
     });
}

Let’s open the console and look at the result generated by the model. The result presents us with an array of possible classifications and the confidence score.

In our project, let’s show the first/the most confident result. To do so, we can edit the message.innerHTML in function userImageUploaded() to below so that the text will change to the prediction once it is classified.

function userImageUploaded() {
     message.innerHTML = “Image was loaded!”

     classifier.classify(img, (err, results) => {
         console.log(results);
         message.innerHTML = `I think it's a ${results[0].label}!`
     });
}

Once you refresh the Show panel and re-upload the image, the browser will tell you the image you uploaded is a hamster (or whatever the ML model thinks)!

How cool is that!?

Step 4. Announce the Result (Optional)

Now we have implemented the ImageClassifer. Why don’t we add a bit of magic to the project by having the browser announce the classification audibly?

At the end of script.js, Erik has included a function called speak(text) which can extract the text message and have the synth to announce it. What we need is to add a line in the userImageUploaded() function, as below.

function userImageUploaded() {
     message.innerHTML = “Image was loaded!”

     classifier.classify(img, (err, results) => {
         console.log(results);
         message.innerHTML = `I think it's a ${results[0].label}!`
         speak(`I think it’s a ${results[0].label}!`)  
     });
}

Refresh the project and wait for the browser announcement…

“I think it’s a hamster!”

Woo-hoo! Now you have completed the first project with the ImageClassifer model using ML5.js! Feel free to experiment and test the model capability by uploading different images and reviewing the full ml5.js documentation to see extended use.

Ran Into Any Issues? No Worries.

You can always review the workshop video on Youtube: https://www.youtube.com/watch?v=BRs1xouy-k4

In addition, Erik has provided the finished file that you can access at: https://glitch.com/~ml5-workshop-image-finished

Conclusion & Next Step

We hope you had fun and gained some new perspectives and ideas to bring back to your own practice. And of course, a big shout out to our brilliant host Erik Katerborg who showed us the potential and fun creative applications of doing Machine Learning in the browser using ML5.

We’d love to hear your key takeaways in a reply or a post if you’re willing to share! Please leave a like and share with your friends if you found the article helpful!

Links to Workshop Materials

To rewatch the workshop, please head over to our Youtube Channel where we have uploaded the recording:

This is Erik’s Glitch account where you can find all his projects: https://glitch.com/@KokoDoko.

Again, you can always access the finished project:

ImageClassifier

Helpful Resources

Beginners guide to ML5 by the Coding Train — https://thecodingtrain.com/learning/ml5/

Made with Tensorflow.js Youtube channel — https://www.youtube.com/watch?v=h9i7d4R36Lw&list=PLQY2H8rRoyvzSZZuF0qJpoJxZR1NgzcZw

ML5.js documentation — https://learn.ml5js.org/

Interested in More Workshops?

Stay tuned to our Eventbrite page for upcoming workshops, keynotes, and networking events!

AI x Design Community

Thank you, Erik!

This tutorial is fully based on the workshop developed and hosted by Erik for AIxDesign in November & December 2021. So big shout out to Erik for this amazing work! Erik Katerborg is a creative technologist and lecturer in creative media at the Rotterdam University for Applied Sciences. He is especially interested in making technology and coding more accessible to people with a creative or design background.

To stay in touch, you can connect with Erik on Linkedin, Twitter, or Instagram.

About AIxDesign

AIxDesign is a place to unite practitioners and evolve practices at the intersection of AI/ML and design. We are currently organizing monthly virtual events (like this one), sharing content, exploring collaborative projects, and developing fruitful partnerships.

To stay in the loop, follow us on Instagram, Linkedin, or subscribe to our monthly newsletter to capture it all in your inbox. You can now also find us at aixdesign.co.